Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      The current schedulers in Hadoop all examine a single job on every heartbeat when choosing which tasks to assign, choosing the job based on FIFO or fair sharing. There are inherent limitations to this approach. For example, if the job at the front of the queue is small (e.g. 10 maps, in a cluster of 100 nodes), then on average it will launch only one local map on the first 10 heartbeats while it is at the head of the queue. This leads to very poor locality for small jobs. Instead, we need a more "global" view of scheduling that can look at multiple jobs. To resolve the locality problem, we will use the following algorithm:

      • If the job at the head of the queue has no node-local task to launch, skip it and look through other jobs.
      • If a job has waited at least T1 seconds while being skipped, also allow it to launch rack-local tasks.
      • If a job has waited at least T2 > T1 seconds, also allow it to launch off-rack tasks.
        This algorithm improves locality while bounding the delay that any job experiences in launching a task.

      It turns out that whether waiting is useful depends on how many tasks are left in the job - the probability of getting a heartbeat from a node with a local task - and on whether the job is CPU or IO bound. Thus there may be logic for removing the wait on the last few tasks in the job.

      As a related issue, once we allow global scheduling, we can launch multiple tasks per heartbeat, as in HADOOP-3136. The initial implementation of HADOOP-3136 adversely affected performance because it only launched multiple tasks from the same job, but with the wait rule above, we will only do this for jobs that are allowed to launch non-local tasks.

      1. fs-global-v0.patch
        67 kB
        Matei Zaharia
      2. HADOOP-4667_api.patch
        5 kB
        Arun C Murthy
      3. hadoop-4667-v1.patch
        56 kB
        Matei Zaharia
      4. hadoop-4667-v1b.patch
        56 kB
        Matei Zaharia
      5. hadoop-4667-v2.patch
        55 kB
        Matei Zaharia
      6. mapreduce-548.patch
        56 kB
        Matei Zaharia
      7. mapreduce-548-v1.patch
        58 kB
        Matei Zaharia
      8. mapreduce-548-v2.patch
        59 kB
        Matei Zaharia
      9. mapreduce-548-v3.patch
        59 kB
        Matei Zaharia
      10. mapreduce-548-v4.patch
        59 kB
        Matei Zaharia

        Issue Links

          Activity

          Hide
          Matei Zaharia added a comment -

          Here's a preliminary version of this patch. It includes the patches for HADOOP-4789 and https://issues.apache.org/jira/browse/HADOOP-4665 because it depends on those. This may make it confusing to read but I will post simpler versions once those patches are in. The code of interest here is really just in assignTasks and getAllowedLocalityLevel in FairScheduler.java. In addition, doing this requires a change to the JobInProgress API to have an obtainNewMapTask version that takes a locality level (distance up the topology). This was already used internally in findMapTask but there is now a package-visible method that exposes it to the scheduler.

          Show
          Matei Zaharia added a comment - Here's a preliminary version of this patch. It includes the patches for HADOOP-4789 and https://issues.apache.org/jira/browse/HADOOP-4665 because it depends on those. This may make it confusing to read but I will post simpler versions once those patches are in. The code of interest here is really just in assignTasks and getAllowedLocalityLevel in FairScheduler.java. In addition, doing this requires a change to the JobInProgress API to have an obtainNewMapTask version that takes a locality level (distance up the topology). This was already used internally in findMapTask but there is now a package-visible method that exposes it to the scheduler.
          Hide
          Arun C Murthy added a comment -

          I'm interested in how you decide values for dataLocalWait and rackLocalWait...

          OTOH, it seems we could use the notion of 'missed opportunities to schedule' rather in-lieu of dataLocalWait/rackLocalWait ... for e.g. we could just use the number of tasktrackers which were rejected by the Job since they didn't have the requisite data etc. Thoughts?

          Show
          Arun C Murthy added a comment - I'm interested in how you decide values for dataLocalWait and rackLocalWait... OTOH, it seems we could use the notion of 'missed opportunities to schedule' rather in-lieu of dataLocalWait/rackLocalWait ... for e.g. we could just use the number of tasktrackers which were rejected by the Job since they didn't have the requisite data etc. Thoughts?
          Hide
          Matei Zaharia added a comment - - edited

          How to set the waits is an interesting question. I set them to be times rather than number of tasktrackers so that they are very easy for an administrator to understand (if they want some kind of guarantee about response time) and so that you don't need to take into account number of nodes in your cluster to decide what is a reasonable number. However, how long to set them for depends on several factors:

          • How much you weigh throughput vs response time. If you care only about throughput, it's generally better to wait longer.
          • Percent of nodes that have local data for the job. If you are down to 1-2 map tasks left to launch, then maybe the expected wait time until you receive a local heartbeat is quite long and you might as well launch non-locally right away.
          • Nature of tasks. If you have a task that is CPU-heavy, then there's little or no loss in response time from launching it non-locally.

          Another thing we've thought about is what to do if there's a "hotspot" node that everyone wants to run on. In this case, setting the waits too high is a bad idea, because you'll end up with a lot of tasks waiting on the hotspot node and with other nodes being underutilized. One interesting question though is which tasks to launch on the hotspot node. If you have an IO-bound job where each task takes 20s to process a block, while another job is more CPU-heavy and takes 60s to process a block, then you want to run the IO-bound job locally and the CPU-bound job non-locally. The reason is that in the time it takes to run one task from the CPU-heavy job, you could've run 3 tasks from the IO-bound one and saved sending those 3 blocks across the network as well as saved response time.

          I haven't included anything to deal with this case or with setting the waits in general in this patch, because we found that 10-15 second waits work well for dataLocalWait and then 20-25s work well for rackLocalWait. However, in a future patch, it might be worthwhile to look at some task statistics to determine IO rate for each job and identify the CPU-bound ones, then lower the waits on those ones so that they go to non-hotspot nodes.

          Show
          Matei Zaharia added a comment - - edited How to set the waits is an interesting question. I set them to be times rather than number of tasktrackers so that they are very easy for an administrator to understand (if they want some kind of guarantee about response time) and so that you don't need to take into account number of nodes in your cluster to decide what is a reasonable number. However, how long to set them for depends on several factors: How much you weigh throughput vs response time. If you care only about throughput, it's generally better to wait longer. Percent of nodes that have local data for the job. If you are down to 1-2 map tasks left to launch, then maybe the expected wait time until you receive a local heartbeat is quite long and you might as well launch non-locally right away. Nature of tasks. If you have a task that is CPU-heavy, then there's little or no loss in response time from launching it non-locally. Another thing we've thought about is what to do if there's a "hotspot" node that everyone wants to run on. In this case, setting the waits too high is a bad idea, because you'll end up with a lot of tasks waiting on the hotspot node and with other nodes being underutilized. One interesting question though is which tasks to launch on the hotspot node. If you have an IO-bound job where each task takes 20s to process a block, while another job is more CPU-heavy and takes 60s to process a block, then you want to run the IO-bound job locally and the CPU-bound job non-locally. The reason is that in the time it takes to run one task from the CPU-heavy job, you could've run 3 tasks from the IO-bound one and saved sending those 3 blocks across the network as well as saved response time. I haven't included anything to deal with this case or with setting the waits in general in this patch, because we found that 10-15 second waits work well for dataLocalWait and then 20-25s work well for rackLocalWait. However, in a future patch, it might be worthwhile to look at some task statistics to determine IO rate for each job and identify the CPU-bound ones, then lower the waits on those ones so that they go to non-hotspot nodes.
          Hide
          Arun C Murthy added a comment -

          I set them to be times rather than number of tasktrackers so that they are very easy for an administrator to understand (if they want some kind of guarantee about response time) and so that you don't need to take into account number of nodes in your cluster to decide what is a reasonable number.

          Hmm... sorry I should have been more clear.

          I propose we use a fraction of the cluster-size (i.e. total no. of tasktrackers in the sytem via ClusterStatus) rather than time. For e.g. we could say that dataLocalWait is equivalent to 100% of cluster size (which implies 1 round of heartbeats i.e. 5s for 500 nodes, 10s for 1000 nodes etc.) and 200% of cluster-size (i.e. 10s for 500, 20s for 1000). This will keep it relatively simple and tractable. Thoughts?

          Show
          Arun C Murthy added a comment - I set them to be times rather than number of tasktrackers so that they are very easy for an administrator to understand (if they want some kind of guarantee about response time) and so that you don't need to take into account number of nodes in your cluster to decide what is a reasonable number. Hmm... sorry I should have been more clear. I propose we use a fraction of the cluster-size (i.e. total no. of tasktrackers in the sytem via ClusterStatus) rather than time. For e.g. we could say that dataLocalWait is equivalent to 100% of cluster size (which implies 1 round of heartbeats i.e. 5s for 500 nodes, 10s for 1000 nodes etc.) and 200% of cluster-size (i.e. 10s for 500, 20s for 1000). This will keep it relatively simple and tractable. Thoughts?
          Hide
          Matei Zaharia added a comment -

          That sounds like a good way to do it. What is the typical inter-heartbeat interval on a large cluster? And does it change if the nodes are busy vs if they're idle? I also imagine that we can just look at the heartbeat interval value (which the master knows) and add a fudge factor to that and set that as the wait time, thus avoiding having to count trackers.

          Show
          Matei Zaharia added a comment - That sounds like a good way to do it. What is the typical inter-heartbeat interval on a large cluster? And does it change if the nodes are busy vs if they're idle? I also imagine that we can just look at the heartbeat interval value (which the master knows) and add a fudge factor to that and set that as the wait time, thus avoiding having to count trackers.
          Hide
          Arun C Murthy added a comment -

          I also imagine that we can just look at the heartbeat interval value (which the master knows) and add a fudge factor to that and set that as the wait time, thus avoiding having to count trackers.

          I'd argue it's easier and more direct to count trackers rather than try and convert it to a timestamp, at the very least it will save you a few million gettimeofday calls...

          Show
          Arun C Murthy added a comment - I also imagine that we can just look at the heartbeat interval value (which the master knows) and add a fudge factor to that and set that as the wait time, thus avoiding having to count trackers. I'd argue it's easier and more direct to count trackers rather than try and convert it to a timestamp, at the very least it will save you a few million gettimeofday calls...
          Hide
          Matei Zaharia added a comment -

          That sounds like it should work and would be easier to code. The only caveat is that we have to be careful in the future if someone changes the heartbeat logic to e.g. reduce the heartbeat rate if the node is busy... hopefully this is not an issue since we just reviewed that.

          Show
          Matei Zaharia added a comment - That sounds like it should work and would be easier to code. The only caveat is that we have to be careful in the future if someone changes the heartbeat logic to e.g. reduce the heartbeat rate if the node is busy... hopefully this is not an issue since we just reviewed that.
          Hide
          Matei Zaharia added a comment -

          Also some notes on the implementation status.. I've been testing and improving a Hadoop 0.17 version of this patch extensively at Facebook. I'll post updates to the patches on here when that's more or less done and I have a chance to port them forward.

          Show
          Matei Zaharia added a comment - Also some notes on the implementation status.. I've been testing and improving a Hadoop 0.17 version of this patch extensively at Facebook. I'll post updates to the patches on here when that's more or less done and I have a chance to port them forward.
          Hide
          Arun C Murthy added a comment -

          Here is a simple api proposal I've attached:

          JobInProgress.addMapsSchedulingOpportunity() which is called at top of the scheduling loop before calling any of the obtain*MapTask methods.

          A JobInProgress.getNumMissedMapsSchedulingOpportunities() method to get the no. of missed opportunities which can be used for global scheduling. We can use the same for HADOOP-4981 and other such areas...

          Thoughts?

          Show
          Arun C Murthy added a comment - Here is a simple api proposal I've attached: JobInProgress.addMapsSchedulingOpportunity() which is called at top of the scheduling loop before calling any of the obtain*MapTask methods. A JobInProgress.getNumMissedMapsSchedulingOpportunities() method to get the no. of missed opportunities which can be used for global scheduling. We can use the same for HADOOP-4981 and other such areas... Thoughts?
          Hide
          Devaraj Das added a comment -

          Arun (as Hemanth and I discussed offline with you), the direction of this seems good but I'd not set up this interface between the Scheduler and the JobInProgress. It seems more natural for the Scheduler to keep track of which jobs missed an opportunity in the current cycle and the corresponding overall counts (maybe through a utility class), rather than embedding that information within the job. Thoughts?

          Show
          Devaraj Das added a comment - Arun (as Hemanth and I discussed offline with you), the direction of this seems good but I'd not set up this interface between the Scheduler and the JobInProgress. It seems more natural for the Scheduler to keep track of which jobs missed an opportunity in the current cycle and the corresponding overall counts (maybe through a utility class), rather than embedding that information within the job. Thoughts?
          Hide
          Matei Zaharia added a comment -

          I agree with Devaraj on this. The JobInProgress class should be agnostic to scheduling algorithm details - it's already doing way too much stuff. To keep track of per-job state, each scheduler can have an internal data structure. The fair scheduler has a JobInfo class for this already, and I believe the capacity scheduler has something similar too.

          I think that ideally JobInProgress shouldn't be much more than a data structure letting you access useful info about the tasks in the job efficiently. The fact that it also does some scheduling (map locality and keeping track of speculative tasks) has led to a number of problems (schedulers scanning JIP fields directly to handle speculation and preemption, data structures not being maintained if speculation is off (HADOOP-4623), no way to know if a job can run a task on a node without calling obtainNew*Task and seeing if it returns null, a growing list of parameters to obtainNewMapTask, etc).

          Show
          Matei Zaharia added a comment - I agree with Devaraj on this. The JobInProgress class should be agnostic to scheduling algorithm details - it's already doing way too much stuff. To keep track of per-job state, each scheduler can have an internal data structure. The fair scheduler has a JobInfo class for this already, and I believe the capacity scheduler has something similar too. I think that ideally JobInProgress shouldn't be much more than a data structure letting you access useful info about the tasks in the job efficiently. The fact that it also does some scheduling (map locality and keeping track of speculative tasks) has led to a number of problems (schedulers scanning JIP fields directly to handle speculation and preemption, data structures not being maintained if speculation is off ( HADOOP-4623 ), no way to know if a job can run a task on a node without calling obtainNew*Task and seeing if it returns null, a growing list of parameters to obtainNewMapTask, etc).
          Hide
          Arun C Murthy added a comment -

          The JobInProgress class should be agnostic to scheduling algorithm details - it's already doing way too much stuff.

          I agree, however that isn't something we can quickly fix. We already have a lot of scheduling information in the JobInProgress - locality-related caches etc. [1]

          To keep track of per-job state, each scheduler can have an internal data structure. The fair scheduler has a JobInfo class for this already, and I believe the capacity scheduler has something similar too.

          Given [1] this seems like we have the worst of both worlds - scheduling state in both JobInProgress and the schedulers, and worse - the exact same information maintained by all schedulers.

          Hence my proposal to just keep it in JobInProgress for now...

          Meanwhile, I'm willing to hear a proposal for a Scheduler independent place to store job-related information, say JobInfo?

          Show
          Arun C Murthy added a comment - The JobInProgress class should be agnostic to scheduling algorithm details - it's already doing way too much stuff. I agree, however that isn't something we can quickly fix. We already have a lot of scheduling information in the JobInProgress - locality-related caches etc. [1] To keep track of per-job state, each scheduler can have an internal data structure. The fair scheduler has a JobInfo class for this already, and I believe the capacity scheduler has something similar too. Given [1] this seems like we have the worst of both worlds - scheduling state in both JobInProgress and the schedulers, and worse - the exact same information maintained by all schedulers. Hence my proposal to just keep it in JobInProgress for now... Meanwhile, I'm willing to hear a proposal for a Scheduler independent place to store job-related information, say JobInfo?
          Hide
          Matei Zaharia added a comment - - edited

          Having some kind of SchedulingInfo data structure associated with each job might be a good start towards separating scheduling stuff from non-scheduling stuff and ultimately having a common scheduler codebase. It could be a field within the JobInProgress, and maybe schedulers would be allowed to extend the base class to have their own info attributes. Is this the kind of thing you're proposing?

          I'm still somewhat wary about making obtainNewMapTask sometimes not return tasks due to this scheduling opportunity stuff though. It seems like the more things we add to it, the harder it will be to break the cycle and switch to a saner API (hasMapTask and createMapTask for example). Furthermore, once a technique like this is in the JobInProgress class, it's hard to try out other methods for achieving the same goal. One of the nicer things about having the scheduler API is that although it makes the codebase more fragmented, it's enabled us to experiment with stuff like this. As a concrete example, once this basic patch is finished, I want to try a refinement for dealing with hotspot nodes that will launch IO-intensive tasks preferentially on those nodes to maximize the rate of local IO. Would it make sense to be working in JobInProgress for that? So I'd prefer if there was provide this functionality to all the schedulers without ingraining it in JobInProgress, or at least without blocking the road towards changes to this policy. Perhaps figuring out how to split up obtainNewMapTask into something generic that everyone can use (give me a task now) and something smarter (count attempts and handle the wait for me) and perhaps even a version that just says whether there is a task with the given locality level without initializing it would be possible without significant code changes. Does that make sense or do you think it's better to leave the API exactly the same and do this by default?

          Show
          Matei Zaharia added a comment - - edited Having some kind of SchedulingInfo data structure associated with each job might be a good start towards separating scheduling stuff from non-scheduling stuff and ultimately having a common scheduler codebase. It could be a field within the JobInProgress, and maybe schedulers would be allowed to extend the base class to have their own info attributes. Is this the kind of thing you're proposing? I'm still somewhat wary about making obtainNewMapTask sometimes not return tasks due to this scheduling opportunity stuff though. It seems like the more things we add to it, the harder it will be to break the cycle and switch to a saner API (hasMapTask and createMapTask for example). Furthermore, once a technique like this is in the JobInProgress class, it's hard to try out other methods for achieving the same goal. One of the nicer things about having the scheduler API is that although it makes the codebase more fragmented, it's enabled us to experiment with stuff like this. As a concrete example, once this basic patch is finished, I want to try a refinement for dealing with hotspot nodes that will launch IO-intensive tasks preferentially on those nodes to maximize the rate of local IO. Would it make sense to be working in JobInProgress for that? So I'd prefer if there was provide this functionality to all the schedulers without ingraining it in JobInProgress, or at least without blocking the road towards changes to this policy. Perhaps figuring out how to split up obtainNewMapTask into something generic that everyone can use (give me a task now) and something smarter (count attempts and handle the wait for me) and perhaps even a version that just says whether there is a task with the given locality level without initializing it would be possible without significant code changes. Does that make sense or do you think it's better to leave the API exactly the same and do this by default?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Matei, I just started looking at this JIRA and and your preliminary approach, but the first patch that you uploaded doesn't apply cleanly any longer on trunk. Agree that the approach is still being discussed upon, but can you please upload an updated patch so that it can help me in getting started? Thanks.

          Show
          Vinod Kumar Vavilapalli added a comment - Matei, I just started looking at this JIRA and and your preliminary approach, but the first patch that you uploaded doesn't apply cleanly any longer on trunk. Agree that the approach is still being discussed upon, but can you please upload an updated patch so that it can help me in getting started? Thanks.
          Hide
          Matei Zaharia added a comment -

          I'll do this sometime in the next few days, since I have a couple of things on my queue before them. We have an updated patch for Hadoop 0.17 at Facebook, so there will be a number of improvements. If you want to try this patch in the meantime though, try reverting to revision 730768 (first week of January) and it should apply to that.

          Show
          Matei Zaharia added a comment - I'll do this sometime in the next few days, since I have a couple of things on my queue before them. We have an updated patch for Hadoop 0.17 at Facebook, so there will be a number of improvements. If you want to try this patch in the meantime though, try reverting to revision 730768 (first week of January) and it should apply to that.
          Hide
          Matei Zaharia added a comment -

          Here is an updated version of this patch. It includes some bug fixes from testing and it is applies against the current trunk. Still haven't take into account some of Arun's comments though.

          Show
          Matei Zaharia added a comment - Here is an updated version of this patch. It includes some bug fixes from testing and it is applies against the current trunk. Still haven't take into account some of Arun's comments though.
          Hide
          Matei Zaharia added a comment -

          Here is an update to this patch to take into account the changes I made to the preemption patch (HADOOP-4665).

          Show
          Matei Zaharia added a comment - Here is an update to this patch to take into account the changes I made to the preemption patch ( HADOOP-4665 ).
          Hide
          Matei Zaharia added a comment -

          Some TODOs left:

          • Switch from time-based waits to heartbeat counting
          • Add a parameter to allow launching multiple tasks per heartbeat as in HADOOP-3136.
          Show
          Matei Zaharia added a comment - Some TODOs left: Switch from time-based waits to heartbeat counting Add a parameter to allow launching multiple tasks per heartbeat as in HADOOP-3136 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12399790/hadoop-4667-v1b.patch
          against trunk revision 742171.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12399790/hadoop-4667-v1b.patch against trunk revision 742171. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3814/console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          Updating patch to use latest changes to HADOOP-4665.

          Show
          Matei Zaharia added a comment - Updating patch to use latest changes to HADOOP-4665 .
          Hide
          Chris Douglas added a comment -

          The patch no longer applies to trunk.

          Show
          Chris Douglas added a comment - The patch no longer applies to trunk.
          Hide
          Matei Zaharia added a comment -

          I've been porting this JIRA to trunk now that HADOOP-4665 is in, and I tried making it use missed scheduling opportunities rather than time waitd in the process. However, I discovered a problem with that approach. Suppose that the cluster is full of long-running tasks except for 1 slot, and that our waiting job doesn't have local data on this slot. If we count missed scheduling opportunities and wait until we've seen as many as the total number of nodes, then we'll wait for (numNodes * heartbeatInterval) seconds, which is a very long time. On the other hand, setting the threshold to something smaller won't work in the case where the cluster is mostly idle. The problem is that the number of scheduling opportunities you get per second depends on the nature of tasks running in the cluster.

          Therefore, I'm going to switch this patch back to counting time so that we have control over the amount of waiting. There will be a single call to System.currentTimeMillis at the start of the scheduler's assignTasks method. Owen, does one call per assignTasks have any performance impact in your experience? I imagine that even logging data through log4j causes gettimeofday to be invoked.

          Show
          Matei Zaharia added a comment - I've been porting this JIRA to trunk now that HADOOP-4665 is in, and I tried making it use missed scheduling opportunities rather than time waitd in the process. However, I discovered a problem with that approach. Suppose that the cluster is full of long-running tasks except for 1 slot, and that our waiting job doesn't have local data on this slot. If we count missed scheduling opportunities and wait until we've seen as many as the total number of nodes, then we'll wait for (numNodes * heartbeatInterval) seconds, which is a very long time. On the other hand, setting the threshold to something smaller won't work in the case where the cluster is mostly idle. The problem is that the number of scheduling opportunities you get per second depends on the nature of tasks running in the cluster. Therefore, I'm going to switch this patch back to counting time so that we have control over the amount of waiting. There will be a single call to System.currentTimeMillis at the start of the scheduler's assignTasks method. Owen, does one call per assignTasks have any performance impact in your experience? I imagine that even logging data through log4j causes gettimeofday to be invoked.
          Hide
          Matei Zaharia added a comment -

          (By the way, another solution I considered is to increment skip counts on every heartbeat, even if there are no free slots to assign tasks into during that heartbeat. But this seems silly - you have to iterate through the whole job list every time just to avoid one gettimeofday call.)

          Show
          Matei Zaharia added a comment - (By the way, another solution I considered is to increment skip counts on every heartbeat, even if there are no free slots to assign tasks into during that heartbeat. But this seems silly - you have to iterate through the whole job list every time just to avoid one gettimeofday call.)
          Hide
          Matei Zaharia added a comment -

          Here is a new patch for this issue for trunk. It includes multiple unit tests both for delay scheduling and for assigning multiple tasks/heartbeat.

          Show
          Matei Zaharia added a comment - Here is a new patch for this issue for trunk. It includes multiple unit tests both for delay scheduling and for assigning multiple tasks/heartbeat.
          Hide
          Matei Zaharia added a comment -

          A note on how the delay scheduling is activated in this patch. I chose to make the amount of time waited for a node-local task and then for a rack-local task to be 1.5x the current heartbeat interval each. This emulates the automatic scaling with heartbeat interval that was suggested. However, it is also possible to configure the delay manually through mapred.fairscheduler.locality.delay, and to disable it by setting this parameter to 0. A value of -1 for this parameter means "compute the delay based on heartbeat interval". That is the default.

          Show
          Matei Zaharia added a comment - A note on how the delay scheduling is activated in this patch. I chose to make the amount of time waited for a node-local task and then for a rack-local task to be 1.5x the current heartbeat interval each. This emulates the automatic scaling with heartbeat interval that was suggested. However, it is also possible to configure the delay manually through mapred.fairscheduler.locality.delay, and to disable it by setting this parameter to 0. A value of -1 for this parameter means "compute the delay based on heartbeat interval". That is the default.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12412446/mapreduce-548.patch
          against trunk revision 790543.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12412446/mapreduce-548.patch against trunk revision 790543. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/339/console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          Holy test failures, Batman. Sadly they seem to be unrelated to this patch (e.g. the streaming ones are from MAPREDUCE-699). Resubmitting patch because the capacity scheduler queue test was fixed this morning.

          Show
          Matei Zaharia added a comment - Holy test failures, Batman. Sadly they seem to be unrelated to this patch (e.g. the streaming ones are from MAPREDUCE-699 ). Resubmitting patch because the capacity scheduler queue test was fixed this morning.
          Hide
          Matei Zaharia added a comment -

          Added a new unit test to verify that we balance load across nodes when the cluster is underloaded.

          Show
          Matei Zaharia added a comment - Added a new unit test to verify that we balance load across nodes when the cluster is underloaded.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12412446/mapreduce-548.patch
          against trunk revision 790918.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12412446/mapreduce-548.patch against trunk revision 790918. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/347/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12412509/mapreduce-548-v1.patch
          against trunk revision 790971.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12412509/mapreduce-548-v1.patch against trunk revision 790971. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/348/console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          Test failures still appear unrelated.

          Show
          Matei Zaharia added a comment - Test failures still appear unrelated.
          Hide
          Matei Zaharia added a comment -

          I updated the patch slightly to fix a deadlock that could occur with the auto-computing of the locality delay. Computing the next heartbeat interval requires a lock on the JobTracker. I was calling this while holding a lock on the FairScheduler, leading to a (FairScheduler -> JobTracker) locking order. However, assignTasks locks the JobTracker before the FairScheduler, so this could cause a deadlock. The solution was to move the auto-computing code outside the FairScheduler lock.

          I also added a cap on the auto-computed locality delay of 15 seconds, because heartbeat intervals in large clusters can be 30-60 seconds. With this new cap there should still be no problem finding a rack-local slot if one exists.

          Show
          Matei Zaharia added a comment - I updated the patch slightly to fix a deadlock that could occur with the auto-computing of the locality delay. Computing the next heartbeat interval requires a lock on the JobTracker. I was calling this while holding a lock on the FairScheduler, leading to a (FairScheduler -> JobTracker) locking order. However, assignTasks locks the JobTracker before the FairScheduler, so this could cause a deadlock. The solution was to move the auto-computing code outside the FairScheduler lock. I also added a cap on the auto-computed locality delay of 15 seconds, because heartbeat intervals in large clusters can be 30-60 seconds. With this new cap there should still be no problem finding a rack-local slot if one exists.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12412935/mapreduce-548-v2.patch
          against trunk revision 792704.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12412935/mapreduce-548-v2.patch against trunk revision 792704. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/370/console This message is automatically generated.
          Hide
          Tom White added a comment -

          I had a question about the latest patch:

          • You have changed obtainNewLocalMapTask() and obtainNewNonLocalMapTask() in JobInProgress to delegate to obtainNewMapTask(). Is the check for !tasksInited.get() (from the former two) equivalent to status.getRunState() != JobStatus.RUNNING from the latter?
          Show
          Tom White added a comment - I had a question about the latest patch: You have changed obtainNewLocalMapTask() and obtainNewNonLocalMapTask() in JobInProgress to delegate to obtainNewMapTask(). Is the check for !tasksInited.get() (from the former two) equivalent to status.getRunState() != JobStatus.RUNNING from the latter?
          Hide
          Arun C Murthy added a comment -

          Matei, can you please ensure that the checks Tom pointed out are put back in?

          Show
          Arun C Murthy added a comment - Matei, can you please ensure that the checks Tom pointed out are put back in?
          Hide
          Matei Zaharia added a comment -

          I've added those checks back. Arun, does this mean that checking for status.getRunState() != JobStatus.RUNNING instead of !tasksInited.get() in obtainNewMapTask is a bug? I removed the checks because I assumed that this one would also catch the tasks-not-inited case. I can also change the check in obtainNewMapTask to use !tasksInited.get() if necessary.

          Show
          Matei Zaharia added a comment - I've added those checks back. Arun, does this mean that checking for status.getRunState() != JobStatus.RUNNING instead of !tasksInited.get() in obtainNewMapTask is a bug? I removed the checks because I assumed that this one would also catch the tasks-not-inited case. I can also change the check in obtainNewMapTask to use !tasksInited.get() if necessary.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12413886/mapreduce-548-v3.patch
          against trunk revision 794942.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413886/mapreduce-548-v3.patch against trunk revision 794942. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/408/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Yes, I think it's a good idea to check for tasksInited.get too... Devaraj?

          Show
          Arun C Murthy added a comment - Yes, I think it's a good idea to check for tasksInited.get too... Devaraj?
          Hide
          Devaraj Das added a comment -

          I think what Matei did is okay. Basically, the tasksInited atomic boolean is used to take care of cases where job is failed/killed while initTasks is in progress (HADOOP-4236), and, for getting the setup/cleanup tasks without trying to lock the JobInProgress (HADOOP-5285). I don't think obtainNewMap* methods need to check the tasksInited flag. In those methods, the check for (jobStatus != JobStatus,RUNNING) should be sufficient since these methods lock the JobInProgress object, and, we invoke these methods only after initTasks has run. It will be nice if someone validates this..
          But having the checks for tasksInited in those methods is harmless except for the extra check..

          Show
          Devaraj Das added a comment - I think what Matei did is okay. Basically, the tasksInited atomic boolean is used to take care of cases where job is failed/killed while initTasks is in progress ( HADOOP-4236 ), and, for getting the setup/cleanup tasks without trying to lock the JobInProgress ( HADOOP-5285 ). I don't think obtainNewMap* methods need to check the tasksInited flag. In those methods, the check for (jobStatus != JobStatus,RUNNING) should be sufficient since these methods lock the JobInProgress object, and, we invoke these methods only after initTasks has run. It will be nice if someone validates this.. But having the checks for tasksInited in those methods is harmless except for the extra check..
          Hide
          Matei Zaharia added a comment -

          I can add a check for both jobStatus != JobStatus.RUNNING and taskInited in my single obtainNewMapTask that the others are calling. In the pre-patch code, there was a lot of code duplication and some versions of obtainNewMapTask checked tasksInited while other checked jobStatus.

          Show
          Matei Zaharia added a comment - I can add a check for both jobStatus != JobStatus.RUNNING and taskInited in my single obtainNewMapTask that the others are calling. In the pre-patch code, there was a lot of code duplication and some versions of obtainNewMapTask checked tasksInited while other checked jobStatus.
          Hide
          Devaraj Das added a comment -

          I think you can remove the check for taskInited. This will be true if the JobStatus is RUNNING.

          Show
          Devaraj Das added a comment - I think you can remove the check for taskInited. This will be true if the JobStatus is RUNNING.
          Hide
          Matei Zaharia added a comment -

          Alright, I've removed the check. Here's an updated patch for the current trunk.

          Show
          Matei Zaharia added a comment - Alright, I've removed the check. Here's an updated patch for the current trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12414243/mapreduce-548-v4.patch
          against trunk revision 798239.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414243/mapreduce-548-v4.patch against trunk revision 798239. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/426/console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          This feature has been committed as part of MAPREDUCE-706, so I'm closing this JIRA as a duplicate.

          Show
          Matei Zaharia added a comment - This feature has been committed as part of MAPREDUCE-706 , so I'm closing this JIRA as a duplicate.

            People

            • Assignee:
              Matei Zaharia
              Reporter:
              Matei Zaharia
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development