Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6304

Specifying node labels when submitting MR jobs

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels when submitting MR jobs.

      1. MAPREDUCE-6304.20150410-1.patch
        6 kB
        Naganarasimha G R
      2. MAPREDUCE-6304.20150411-1.patch
        6 kB
        Naganarasimha G R
      3. MAPREDUCE-6304.20150501-1.patch
        21 kB
        Naganarasimha G R
      4. MAPREDUCE-6304.20150510-1.patch
        21 kB
        Naganarasimha G R
      5. MAPREDUCE-6304.20150511-1.patch
        21 kB
        Naganarasimha G R
      6. MAPREDUCE-6304.20150512-1.patch
        22 kB
        Naganarasimha G R
      7. MAPREDUCE-6304-branch-2.7.patch
        22 kB
        Inigo Goiri

        Issue Links

          Activity

          Hide
          john.jian.fang Jian Fang added a comment -

          Link related JIRAs

          Show
          john.jian.fang Jian Fang added a comment - Link related JIRAs
          Hide
          yufeldman Yuliya Feldman added a comment -

          Should be as simple as:
          YarnRunner sets label on ApplicationSubmissionContext for MapReduce applications based on additional configuration property like mapreduce.job.label.

          It is described in https://issues.apache.org/jira/secure/attachment/12654148/LabelBasedScheduling.pdf even though it was not done as such.

          Show
          yufeldman Yuliya Feldman added a comment - Should be as simple as: YarnRunner sets label on ApplicationSubmissionContext for MapReduce applications based on additional configuration property like mapreduce.job.label. It is described in https://issues.apache.org/jira/secure/attachment/12654148/LabelBasedScheduling.pdf even though it was not done as such.
          Hide
          john.jian.fang Jian Fang added a comment -

          mapreduce.job.label may not be enough. There should be at least another parameter such as mapreduce.job.am.label for the application master. For example, on EC2, we don't want to run an application master on a spot instance, but we do allow MR tasks to run on spot instances (otherwise, what is the purpose to use instances?). Furthermore, Application Master is a special Yarn container and MRAppMaster does not run as a YarnChild, right?

          Show
          john.jian.fang Jian Fang added a comment - mapreduce.job.label may not be enough. There should be at least another parameter such as mapreduce.job.am.label for the application master. For example, on EC2, we don't want to run an application master on a spot instance, but we do allow MR tasks to run on spot instances (otherwise, what is the purpose to use instances?). Furthermore, Application Master is a special Yarn container and MRAppMaster does not run as a YarnChild, right?
          Hide
          yufeldman Yuliya Feldman added a comment -

          Jian Fang Definitely it depends whether you want separate label for AM and the rest of the tasks.

          Show
          yufeldman Yuliya Feldman added a comment - Jian Fang Definitely it depends whether you want separate label for AM and the rest of the tasks.
          Hide
          john.jian.fang Jian Fang added a comment -

          YarnRunner is the right place to set the labels for a MR job. However, there is one concern here. If I understand correctly, YarnRunner runs at the job client side, right? There should also be a way to hook in the labels on the server side, i.e., resource manager side. The reason is that many Hadoop users do not understand or set the labels by themselves and they simply rely on the Hadoop platform provider (or system admins for on-premise clusters) to set up the labels for them. I am not sure if this is a general use case, but it is definitely a feature that we need.

          Show
          john.jian.fang Jian Fang added a comment - YarnRunner is the right place to set the labels for a MR job. However, there is one concern here. If I understand correctly, YarnRunner runs at the job client side, right? There should also be a way to hook in the labels on the server side, i.e., resource manager side. The reason is that many Hadoop users do not understand or set the labels by themselves and they simply rely on the Hadoop platform provider (or system admins for on-premise clusters) to set up the labels for them. I am not sure if this is a general use case, but it is definitely a feature that we need.
          Hide
          yufeldman Yuliya Feldman added a comment -

          I do not think labels should be dictated by RM, since different types of jobs may require different types of resources. It is definitely true that cluster admins label nodes so those labels can be used while making a decision from the application prospective.
          It is true that job is submitted from the client, but it is not like you submit different classes of jobs every time, so eventually job owner will know what resources he/she needs for that specific class of jobs.
          The case you described warrants just setting labels in mapred-site.xml on client side and be done. If you submit all sorts of jobs then may be you don't even care about labels.
          Yet another choice is to utilize labels on Queues - this will be set on the cluster so for jobs submitted to a specific queue specific labels will be applied

          Show
          yufeldman Yuliya Feldman added a comment - I do not think labels should be dictated by RM, since different types of jobs may require different types of resources. It is definitely true that cluster admins label nodes so those labels can be used while making a decision from the application prospective. It is true that job is submitted from the client, but it is not like you submit different classes of jobs every time, so eventually job owner will know what resources he/she needs for that specific class of jobs. The case you described warrants just setting labels in mapred-site.xml on client side and be done. If you submit all sorts of jobs then may be you don't even care about labels. Yet another choice is to utilize labels on Queues - this will be set on the cluster so for jobs submitted to a specific queue specific labels will be applied
          Hide
          john.jian.fang Jian Fang added a comment -

          I understand your point from on-premise cluster perspective. However, it is not very practical to manage mapred-site.xml or queue files for users if hadoop is a service in cloud. As a hadoop developer, you should consider both on-premise hadoop cluster and hadoop in cloud.

          There are many many users for a hadoop cloud service. Usually they launch their own hadoop clusters in cloud and control their own queue files or mapred-site.xml. Some of them even run their hadoop jobs on their own gateways that the hadoop platform provider does not have access to. But the hadoop service provider may still want to have a mechanism to set up some global labels for all users to improve their user experiences. For example, a failure of an application master on a spot instance due to the termination of a spot instance will cause more trouble than a failure of one MR task. These types of settings most likely can only be done by hadoop cloud service providers based on their deep knowledge in their own cloud services.

          Or could hadoop provide a mechanism for hadoop providers to extend so that you only need to specify the labels in YarnRunner in Vanilla hadoop?

          Show
          john.jian.fang Jian Fang added a comment - I understand your point from on-premise cluster perspective. However, it is not very practical to manage mapred-site.xml or queue files for users if hadoop is a service in cloud. As a hadoop developer, you should consider both on-premise hadoop cluster and hadoop in cloud. There are many many users for a hadoop cloud service. Usually they launch their own hadoop clusters in cloud and control their own queue files or mapred-site.xml. Some of them even run their hadoop jobs on their own gateways that the hadoop platform provider does not have access to. But the hadoop service provider may still want to have a mechanism to set up some global labels for all users to improve their user experiences. For example, a failure of an application master on a spot instance due to the termination of a spot instance will cause more trouble than a failure of one MR task. These types of settings most likely can only be done by hadoop cloud service providers based on their deep knowledge in their own cloud services. Or could hadoop provide a mechanism for hadoop providers to extend so that you only need to specify the labels in YarnRunner in Vanilla hadoop?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the comments Yuliya Feldman & Jian Fang,

          Definitely it depends whether you want separate label for AM and the rest of the tasks.

          I have started working on this patch and had already considered the 2 kinds of labels as ApplicationSubmissionContext was supporting it and REST has already taken care of it .

          The reason is that many Hadoop users do not understand or set the labels by themselves and they simply rely on the Hadoop platform provider (or system admins for on-premise clusters) to set up the labels for them. I am not sure if this is a general use case, but it is definitely a feature that we need.

          Even though labels should not be dictated by RM, i feel there might be cases as mentioned by Jian Fang where users might not be aware what each label stands for! or users might not want to understand the label as partition (which is more on administrative side in multi tenant scenarios) and they might just require some auto mapping based on user or group. So i think we can come up with some configurations of automapping based on user/group (similar to queues) for labels too.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the comments Yuliya Feldman & Jian Fang , Definitely it depends whether you want separate label for AM and the rest of the tasks. I have started working on this patch and had already considered the 2 kinds of labels as ApplicationSubmissionContext was supporting it and REST has already taken care of it . The reason is that many Hadoop users do not understand or set the labels by themselves and they simply rely on the Hadoop platform provider (or system admins for on-premise clusters) to set up the labels for them. I am not sure if this is a general use case, but it is definitely a feature that we need. Even though labels should not be dictated by RM, i feel there might be cases as mentioned by Jian Fang where users might not be aware what each label stands for! or users might not want to understand the label as partition (which is more on administrative side in multi tenant scenarios) and they might just require some auto mapping based on user or group. So i think we can come up with some configurations of automapping based on user/group (similar to queues) for labels too.
          Hide
          john.jian.fang Jian Fang added a comment -

          Thanks Naganarasimha for your understanding. However, user/group mapping may not work for us since we don't have control of that as a hadoop service provider. I would prefer a plugin mechanism rather than a solution here so that we can extend that for our service. But I think the change for YarnRunner is still needed for hadoop users anyway.

          Show
          john.jian.fang Jian Fang added a comment - Thanks Naganarasimha for your understanding. However, user/group mapping may not work for us since we don't have control of that as a hadoop service provider. I would prefer a plugin mechanism rather than a solution here so that we can extend that for our service. But I think the change for YarnRunner is still needed for hadoop users anyway.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          But I think the change for YarnRunner is still needed for hadoop users anyway.

          yes that true, other discussion is more like an additional requirement, which i think needs attention from Tan, Wangda too.

          Show
          Naganarasimha Naganarasimha G R added a comment - But I think the change for YarnRunner is still needed for hadoop users anyway. yes that true, other discussion is more like an additional requirement, which i think needs attention from Tan, Wangda too.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Just found I'm not in the watcher list, missed some discussions.

          I think Jian Fang's use case is very unique but also interesting, platform provider has no control and knowledge of user's applications, and user doesn't understand about platform's details such as which machine is temporary or not.

          Maybe a possible way to solve this problem is to add a global-am-resource-request setting, which can include node-label-expression, node/rack information. Because as YARN RM, it doesn't know what's the application's running model, it only knows which container is AM or not, which seems enough for your requirements.

          But adding such global-am-resource-request setting can also be dangerous, such as how about a queue is not accessible node-label-expression of global-am-resource-request?

          Show
          leftnoteasy Wangda Tan added a comment - Just found I'm not in the watcher list, missed some discussions. I think Jian Fang 's use case is very unique but also interesting, platform provider has no control and knowledge of user's applications, and user doesn't understand about platform's details such as which machine is temporary or not. Maybe a possible way to solve this problem is to add a global-am-resource-request setting, which can include node-label-expression, node/rack information. Because as YARN RM, it doesn't know what's the application's running model, it only knows which container is AM or not, which seems enough for your requirements. But adding such global-am-resource-request setting can also be dangerous, such as how about a queue is not accessible node-label-expression of global-am-resource-request?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Attaching a patch for this issue

          Show
          Naganarasimha Naganarasimha G R added a comment - Attaching a patch for this issue
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda, As global-am-resource-request can have some caveats which you mention, what if we can have yarn.scheduler.capacity.{queue_path}.default-am-node-label-expression which we can ensure while loading if the labels configured for the queue path exists or not. Or in much more generic way, can we have some ContainerPlacementPolicy interface @ CS which needs to provide which queue and label to be used for a given container, user & group ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda , As global-am-resource-request can have some caveats which you mention, what if we can have yarn.scheduler.capacity.{queue_path}.default-am-node-label-expression which we can ensure while loading if the labels configured for the queue path exists or not. Or in much more generic way, can we have some ContainerPlacementPolicy interface @ CS which needs to provide which queue and label to be used for a given container, user & group ?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12724623/MAPREDUCE-6304.20150410-1.patch
          against trunk revision 7660da9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5389//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724623/MAPREDUCE-6304.20150410-1.patch against trunk revision 7660da9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5389//console This message is automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          correcting the previous patch

          Show
          Naganarasimha Naganarasimha G R added a comment - correcting the previous patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12724652/MAPREDUCE-6304.20150411-1.patch
          against trunk revision 7660da9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5390//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5390//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724652/MAPREDUCE-6304.20150411-1.patch against trunk revision 7660da9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 eclipse:eclipse . The patch failed to build with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5390//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5390//console This message is automatically generated.
          Hide
          john.jian.fang Jian Fang added a comment -

          A more generic way could be to add a decorator to ApplicationSubmissionContext in class ClientRMService so that people can change the ApplicationSubmissionContext in the method submitApplication(). The default decorator from Apache does nothing, but hadoop allows users to use a custom decorator from hadoop configuration.

          Show
          john.jian.fang Jian Fang added a comment - A more generic way could be to add a decorator to ApplicationSubmissionContext in class ClientRMService so that people can change the ApplicationSubmissionContext in the method submitApplication(). The default decorator from Apache does nothing, but hadoop allows users to use a custom decorator from hadoop configuration.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Jian Fang, thanks for the comments but did not get your idea completely. Comment is for the approach taken in the patch or earlier discussion of supporting default label expression for am container ?

          The default decorator from Apache does nothing, but hadoop allows users to use a custom decorator from hadoop configuration.

          Can you please specify which configuration you are referring to ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Jian Fang , thanks for the comments but did not get your idea completely. Comment is for the approach taken in the patch or earlier discussion of supporting default label expression for am container ? The default decorator from Apache does nothing, but hadoop allows users to use a custom decorator from hadoop configuration. Can you please specify which configuration you are referring to ?
          Hide
          john.jian.fang Jian Fang added a comment -

          I mean hadoop could provide a new mechanism such as a decorator for the ApplicationSubmissionContext. When the method submitApplication() in class ClientRMService is called, hadoop decorates the ApplicationSubmissionContext before it calls the following line. For example, manipulates the amLabelExpression.

          rmAppManager.submitApplication(submissionContext,
          System.currentTimeMillis(), user);

          Hadoop could provide a default decorator that does nothing. But users could override the default decorator in yarn-site.xml by a new configuration parameter, for example, "yarn.app.submission.context.decorator.class".

          This new mechanism is not directly related to the change you are making, but it is more generic so that the platform providers could update the ApplicationSubmissionContext in their own ways. Once we have such a new mechanism in place, you do not really need to add anything new to your label code for my use case. Instead, the custom logic will be included in the custom decorator provided by the platform provider. For example, we could provide a decorator to update amLabelExpression in ApplicationSubmissionContext. Other fields of ApplicationSubmissionContext could be changed as well to meet user's needs.

          Show
          john.jian.fang Jian Fang added a comment - I mean hadoop could provide a new mechanism such as a decorator for the ApplicationSubmissionContext. When the method submitApplication() in class ClientRMService is called, hadoop decorates the ApplicationSubmissionContext before it calls the following line. For example, manipulates the amLabelExpression. rmAppManager.submitApplication(submissionContext, System.currentTimeMillis(), user); Hadoop could provide a default decorator that does nothing. But users could override the default decorator in yarn-site.xml by a new configuration parameter, for example, "yarn.app.submission.context.decorator.class". This new mechanism is not directly related to the change you are making, but it is more generic so that the platform providers could update the ApplicationSubmissionContext in their own ways. Once we have such a new mechanism in place, you do not really need to add anything new to your label code for my use case. Instead, the custom logic will be included in the custom decorator provided by the platform provider. For example, we could provide a decorator to update amLabelExpression in ApplicationSubmissionContext. Other fields of ApplicationSubmissionContext could be changed as well to meet user's needs.
          Hide
          john.jian.fang Jian Fang added a comment -

          I created JIRA YARN-3490 for the application decorator proposal.

          Show
          john.jian.fang Jian Fang added a comment - I created JIRA YARN-3490 for the application decorator proposal.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Tan, Wangda, if you have bandwidth can you take a look @ this small patch ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Tan, Wangda , if you have bandwidth can you take a look @ this small patch ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda,Harsh J,Jian He,
          Can any one take a look at this patch ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda , Harsh J , Jian He , Can any one take a look at this patch ?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Naganarasimha G R,
          Just took a look at the patch, some comments about syntax:

          • Beyond AM/Job node-labels setting, should we support Mapper/Reducer setting?
          • When job and AM (or mapper/reducer) set together, AM (or mapper/reducer)'s node-label-expression should overwrite job's setting.
          • If we need support mapper/reducer's node-label-expression. We may need add some changes in MR AM side and some tests.

          Any ideas? Lohit Vijayarenu.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Naganarasimha G R , Just took a look at the patch, some comments about syntax: Beyond AM/Job node-labels setting, should we support Mapper/Reducer setting? When job and AM (or mapper/reducer) set together, AM (or mapper/reducer)'s node-label-expression should overwrite job's setting. If we need support mapper/reducer's node-label-expression. We may need add some changes in MR AM side and some tests. Any ideas? Lohit Vijayarenu .
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review comments Tan, Wangda,

          Beyond AM/Job node-labels setting, should we support Mapper/Reducer setting?

          If we need support mapper/reducer's node-label-expression. We may need add some changes in MR AM side and some tests.

          Well my view should be to support them as i feel in some cases we might feel like mapper can run in any node but may be reducer requires more mem so may be clients might require it to be run in high mem nodes (may be some constraint on mem for the nodes) or some other node constraints.

          When job and AM (or mapper/reducer) set together, AM (or mapper/reducer)'s node-label-expression should overwrite job's setting.

          Well, IIUC then ApplicationSubmissionContext.getNodeLabelExpression() is for mapper/reducer and AM's node label expression is set in ApplicationSubmissionContext.getAMContainerResourceRequest. I could cross verify the same with code in ApplicationMasterService (line number 495) & RMAppManager (line number 378) and so did not get what you meant by the above statement, correct me if my understanding is wrong.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review comments Tan, Wangda , Beyond AM/Job node-labels setting, should we support Mapper/Reducer setting? If we need support mapper/reducer's node-label-expression. We may need add some changes in MR AM side and some tests. Well my view should be to support them as i feel in some cases we might feel like mapper can run in any node but may be reducer requires more mem so may be clients might require it to be run in high mem nodes (may be some constraint on mem for the nodes) or some other node constraints. When job and AM (or mapper/reducer) set together, AM (or mapper/reducer)'s node-label-expression should overwrite job's setting. Well, IIUC then ApplicationSubmissionContext.getNodeLabelExpression() is for mapper/reducer and AM's node label expression is set in ApplicationSubmissionContext.getAMContainerResourceRequest . I could cross verify the same with code in ApplicationMasterService (line number 495) & RMAppManager (line number 378) and so did not get what you meant by the above statement, correct me if my understanding is wrong.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Well my view should be to support them as i feel in some cases we might feel like mapper can run in any node but may be reducer requires more mem so may be clients might require it to be run in high mem nodes (may be some constraint on mem for the nodes) or some other node constraints.

          That's correct, so you agree to add mapper/reducer node-label-expression?

          Well, IIUC then ApplicationSubmissionContext.getNodeLabelExpression() is for mapper/reducer and AM's node label expression is set in ApplicationSubmissionContext.getAMContainerResourceRequest. I could cross verify the same with code in ApplicationMasterService (line number 495) & RMAppManager (line number 378) and so did not get what you meant by the above statement, correct me if my understanding is wrong.

          What I meant is:
          When job.label = x, and don't set am/mapper/reducer label, all containers should get allocated on x
          When job.label = x and am.label = y, don't set mapper/reducer label, am will get allocated on y (overwritten x) but mapper/reducer should get allocated on x.
          When job.label = x, am.label=y, mapper.label=z, am will get allocated on y, mapper will get allocated on z (overwrite x), and reducer get allocated on x.
          This should be existing behavior. And mapper/reducer's label should be added to ResourceRequest (may need modify RMContainerRequestor) instead of AplicationSubmissionContext.

          Show
          leftnoteasy Wangda Tan added a comment - Well my view should be to support them as i feel in some cases we might feel like mapper can run in any node but may be reducer requires more mem so may be clients might require it to be run in high mem nodes (may be some constraint on mem for the nodes) or some other node constraints. That's correct, so you agree to add mapper/reducer node-label-expression? Well, IIUC then ApplicationSubmissionContext.getNodeLabelExpression() is for mapper/reducer and AM's node label expression is set in ApplicationSubmissionContext.getAMContainerResourceRequest. I could cross verify the same with code in ApplicationMasterService (line number 495) & RMAppManager (line number 378) and so did not get what you meant by the above statement, correct me if my understanding is wrong. What I meant is: When job.label = x, and don't set am/mapper/reducer label, all containers should get allocated on x When job.label = x and am.label = y, don't set mapper/reducer label, am will get allocated on y (overwritten x) but mapper/reducer should get allocated on x. When job.label = x, am.label=y, mapper.label=z, am will get allocated on y, mapper will get allocated on z (overwrite x), and reducer get allocated on x. This should be existing behavior. And mapper/reducer's label should be added to ResourceRequest (may need modify RMContainerRequestor ) instead of AplicationSubmissionContext.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for detailed clarification, got what you intended and was planning to do the same way, i.e. only if "mapreduce.job.map.node-label-expression" or "mapreduce.job.reduce.node-label-expression" config are set then modify the ask in RMContainerRequestor as yarn takes care of setting in ApplicationMasterService when no labels specified in resource request for mapper and reducer and in RMAppManager for AM.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for detailed clarification, got what you intended and was planning to do the same way, i.e. only if "mapreduce.job.map.node-label-expression" or "mapreduce.job.reduce.node-label-expression" config are set then modify the ask in RMContainerRequestor as yarn takes care of setting in ApplicationMasterService when no labels specified in resource request for mapper and reducer and in RMAppManager for AM.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda, have incorporated changes for add mapper/reducer node-label-expression also incorported limitations set by YARN-2694(Ensure only single node label specified in ResourceRequest) which limits labels to be supported on only "ANY" ResourceRequests.
          But one query what i have is : Isnt it a serious limitation if labels are supported for only ANY resource requests?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda , have incorporated changes for add mapper/reducer node-label-expression also incorported limitations set by YARN-2694 (Ensure only single node label specified in ResourceRequest) which limits labels to be supported on only "ANY" ResourceRequests. But one query what i have is : Isnt it a serious limitation if labels are supported for only ANY resource requests?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 42s The applied patch generated 16 new checkstyle issues (total was 814, now 830).
          -1 whitespace 0m 2s The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 2m 57s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 8m 40s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 1m 34s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 105m 27s Tests passed in hadoop-mapreduce-client-jobclient.
              154m 49s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-app
            Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java:[line 218]



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729713/MAPREDUCE-6304.20150501-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1b3b9e5
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 42s The applied patch generated 16 new checkstyle issues (total was 814, now 830). -1 whitespace 0m 2s The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 57s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 8m 40s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 1m 34s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 105m 27s Tests passed in hadoop-mapreduce-client-jobclient.     154m 49s   Reason Tests FindBugs module:hadoop-mapreduce-client-app   Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java: [line 218] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729713/MAPREDUCE-6304.20150501-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1b3b9e5 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5482/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          hi Tan, Wangda checkstyle has only valid white space issues, correcting it. Findbugs issues is not so relveant for the variables usage, resubmitting the patch with the white space issue

          Show
          Naganarasimha Naganarasimha G R added a comment - hi Tan, Wangda checkstyle has only valid white space issues, correcting it. Findbugs issues is not so relveant for the variables usage, resubmitting the patch with the white space issue
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 56s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 51s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 32s The applied patch generated 12 new checkstyle issues (total was 814, now 826).
          -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 3m 32s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 10m 48s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 2m 17s Tests passed in hadoop-mapreduce-client-core.
          -1 mapreduce tests 100m 51s Tests failed in hadoop-mapreduce-client-jobclient.
              154m 16s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-app
            Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java:[line 218]
          Failed unit tests hadoop.mapred.TestMiniMRClientCluster
            hadoop.mapred.TestMRTimelineEventHandling
          Timed out tests org.apache.hadoop.mapreduce.TestLargeSort



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729764/MAPREDUCE-6304.20150501-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1b3b9e5
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 51s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 32s The applied patch generated 12 new checkstyle issues (total was 814, now 826). -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 3m 32s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 10m 48s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 2m 17s Tests passed in hadoop-mapreduce-client-core. -1 mapreduce tests 100m 51s Tests failed in hadoop-mapreduce-client-jobclient.     154m 16s   Reason Tests FindBugs module:hadoop-mapreduce-client-app   Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java: [line 218] Failed unit tests hadoop.mapred.TestMiniMRClientCluster   hadoop.mapred.TestMRTimelineEventHandling Timed out tests org.apache.hadoop.mapreduce.TestLargeSort Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729764/MAPREDUCE-6304.20150501-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1b3b9e5 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5483/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R,
          Patch generally LGTM, one minor comment:

          • If specify label="" in mapred-default.xml, which means the job will always set label="" while adding resource-request, but the previously behavior is send "null" instead of empty string. So I suggest to change the default to be "NOT SPECIFIED" (which is not a valid label since it contains space) as default value, and we will replace it by null. Does this make sense to you? Do you have any other ideas on this?

          And could you deploy a few nodes cluster, with 2-3 labels, and run MR job with am.label, mapper.label, reducer.label, etc. that will be very important to make sure everything works well.

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , Patch generally LGTM, one minor comment: If specify label="" in mapred-default.xml, which means the job will always set label="" while adding resource-request, but the previously behavior is send "null" instead of empty string. So I suggest to change the default to be "NOT SPECIFIED" (which is not a valid label since it contains space) as default value, and we will replace it by null. Does this make sense to you? Do you have any other ideas on this? And could you deploy a few nodes cluster, with 2-3 labels, and run MR job with am.label, mapper.label, reducer.label, etc. that will be very important to make sure everything works well.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for reviewing Tan, Wangda,

          If specify label="" in mapred-default.xml, which means the job will always set label="" .... So I suggest to change the default to be "NOT SPECIFIED" (which is not a valid label since it contains space) as default value, and we will replace it by null.

          Good catch, but AFAIK we need not do so much handling and if we just remove <value></value> in mapred-default.xml then the value will be taken as null by xml parser in the Configuration class. while i am testing in the multi node cluster will verify this and then update the patch.

          could you deploy a few nodes cluster, with 2-3 labels, and run MR job with am.label, mapper.label, reducer.label, etc. that will be very important to make sure everything works well.

          Will try to get this tested on monday ( currenty only left with laptop ).
          Test case failures doesn't seem to be related to the patch based on logs.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for reviewing Tan, Wangda , If specify label="" in mapred-default.xml, which means the job will always set label="" .... So I suggest to change the default to be "NOT SPECIFIED" (which is not a valid label since it contains space) as default value, and we will replace it by null. Good catch, but AFAIK we need not do so much handling and if we just remove <value></value> in mapred-default.xml then the value will be taken as null by xml parser in the Configuration class. while i am testing in the multi node cluster will verify this and then update the patch. could you deploy a few nodes cluster, with 2-3 labels, and run MR job with am.label, mapper.label, reducer.label, etc. that will be very important to make sure everything works well. Will try to get this tested on monday ( currenty only left with laptop ). Test case failures doesn't seem to be related to the patch based on logs.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R, remove from mapred-default.xml is not good. mapred-default.xml is a way to document all options we have, how about make its default to be "USE_QUEUE_DEFINED_DEFAULT" or better name, it's not "not specified" or "null" actually, it's using queue defined node label expression. Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , remove from mapred-default.xml is not good. mapred-default.xml is a way to document all options we have, how about make its default to be "USE_QUEUE_DEFINED_DEFAULT" or better name, it's not "not specified" or "null" actually, it's using queue defined node label expression. Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          hi Tan, Wangda,
          I meant actually only deletion of <value></value> for example

          <property>
          <name>mapreduce.job.am.node-label-expression</name>
          <!-- value></value -->
          <description>Overrides mapreduce.job.node-label-expression for application
          master containers
          </description>
          </property>

          which is similar to the existing configs like yarn.ipc.*.factory.class, yarn.resourcemanager.cluster-id,yarn.resourcemanager.ha.rm-ids,yarn.resourcemanager.ha.id ....
          In general your approach is fine but as we can make use of the above approach of configuration did not want to bring in additional default configs. thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - hi Tan, Wangda , I meant actually only deletion of <value></value> for example <property> <name>mapreduce.job.am.node-label-expression</name> <!-- value></value --> <description>Overrides mapreduce.job.node-label-expression for application master containers </description> </property> which is similar to the existing configs like yarn.ipc.*.factory.class, yarn.resourcemanager.cluster-id,yarn.resourcemanager.ha.rm-ids,yarn.resourcemanager.ha.id .... In general your approach is fine but as we can make use of the above approach of configuration did not want to bring in additional default configs. thoughts ?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R, thanks for pointing me about yarn.ipc.*.factory, etc. I think it's important to

          • Not bring in additional unncessary default config
          • Follow what we have in *default.xml
          • Make admin easy to understand

          So I think it's fine to do as what you suggested, but could you please mention in description that, by default the node-label-expression for job is not set, it will use queue's default-node-label-expression.

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , thanks for pointing me about yarn.ipc.*.factory, etc. I think it's important to Not bring in additional unncessary default config Follow what we have in *default.xml Make admin easy to understand So I think it's fine to do as what you suggested, but could you please mention in description that, by default the node-label-expression for job is not set, it will use queue's default-node-label-expression.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda for your comments,
          +1 for mention in description that, by default the node-label-expression for job is not set, it will use queue's default-node-label-expression.. I am getting it tested in cluster setup, will upload the updated patch today.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda for your comments, +1 for mention in description that, by default the node-label-expression for job is not set, it will use queue's default-node-label-expression. . I am getting it tested in cluster setup, will upload the updated patch today.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Updated the description and tested in 2 NM with 3 labels and also specifying different combination of labels

          Show
          Naganarasimha Naganarasimha G R added a comment - Updated the description and tested in 2 NM with 3 labels and also specifying different combination of labels
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 3m 51s The applied patch generated 12 new checkstyle issues (total was 497, now 509).
          -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 2m 53s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 9m 30s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 1m 38s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 108m 46s Tests passed in hadoop-mapreduce-client-jobclient.
              161m 10s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-app
            Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java:[line 218]



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731811/MAPREDUCE-6304.20150510-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4536399
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 3m 51s The applied patch generated 12 new checkstyle issues (total was 497, now 509). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 2m 53s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 9m 30s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 1m 38s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 108m 46s Tests passed in hadoop-mapreduce-client-jobclient.     161m 10s   Reason Tests FindBugs module:hadoop-mapreduce-client-app   Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java: [line 218] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731811/MAPREDUCE-6304.20150510-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4536399 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5699/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Attaching patch with applicable white-space and check style issues fixed

          Show
          Naganarasimha Naganarasimha G R added a comment - Attaching patch with applicable white-space and check style issues fixed
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 30s The applied patch generated 8 new checkstyle issues (total was 497, now 505).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 2m 53s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 8m 58s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 106m 15s Tests passed in hadoop-mapreduce-client-jobclient.
              155m 44s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-app
            Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java:[line 218]



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731906/MAPREDUCE-6304.20150511-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3fa2efc
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 30s The applied patch generated 8 new checkstyle issues (total was 497, now 505). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 53s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 8m 58s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 106m 15s Tests passed in hadoop-mapreduce-client-jobclient.     155m 44s   Reason Tests FindBugs module:hadoop-mapreduce-client-app   Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java: [line 218] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731906/MAPREDUCE-6304.20150511-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3fa2efc checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Naganarasimha G R for working on this and testing, mostly LGTM, could you add the overwriting behavior in mapred-default.xml? (For example, by default is using queue's default-node-label-expression, AM.expression can overwrite job.expression, etc.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Naganarasimha G R for working on this and testing, mostly LGTM, could you add the overwriting behavior in mapred-default.xml? (For example, by default is using queue's default-node-label-expression, AM.expression can overwrite job.expression, etc.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda, have modified the documentation in Mapred-default.xml. Please check.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda , have modified the documentation in Mapred-default.xml. Please check.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 43s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 34s The applied patch generated 8 new checkstyle issues (total was 503, now 511).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 40s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 2m 53s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 9m 0s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 105m 52s Tests passed in hadoop-mapreduce-client-jobclient.
              155m 38s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-app
            Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java:[line 218]



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732135/MAPREDUCE-6304.20150512-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3d28611
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 34s The applied patch generated 8 new checkstyle issues (total was 503, now 511). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 40s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 53s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 9m 0s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 105m 52s Tests passed in hadoop-mapreduce-client-jobclient.     155m 38s   Reason Tests FindBugs module:hadoop-mapreduce-client-app   Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression; locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of time Unsynchronized access at RMContainerAllocator.java: [line 218] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732135/MAPREDUCE-6304.20150512-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3d28611 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5708/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks update, Naganarasimha G R, +1 for latest patch.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks update, Naganarasimha G R , +1 for latest patch.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda
          One suggestion for this patch : labels will be applicable for only for Any resource requests, so would it be better to have a config, based on which rack local and node specific will not be submitted and only Any resource requests with label will be submitted ? This will ensure that MR tasks run only on the labelled nodes.
          If the existing support to labels are sufficient then can we push this in ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda One suggestion for this patch : labels will be applicable for only for Any resource requests, so would it be better to have a config, based on which rack local and node specific will not be submitted and only Any resource requests with label will be submitted ? This will ensure that MR tasks run only on the labelled nodes. If the existing support to labels are sufficient then can we push this in ?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R,
          Actually now CS will check label-expression of ANY resource request matches labels on nodes before allocating rack/node local requests. IAW, node-label-expression of rack/node local request will be ignored and will be treated as same as ANY request. To reply your concern:
          MR tasks run only on the labeled nodes.

          Since this changes MR configuration, I will wait for another couple of days before commit this.

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , Actually now CS will check label-expression of ANY resource request matches labels on nodes before allocating rack/node local requests. IAW, node-label-expression of rack/node local request will be ignored and will be treated as same as ANY request. To reply your concern: MR tasks run only on the labeled nodes. Since this changes MR configuration, I will wait for another couple of days before commit this. Thanks,
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda for clarifying my query, and ok for waiting for couple days for committing this.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda for clarifying my query, and ok for waiting for couple days for committing this.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R, I think I made a mistake in previous comment, rack/node local resource request is not allowed to specify label-expression for now. See SchedulerUtils.validateResourceRequest, I think you need to add a check in MR side, if the resourceName is not ANY, don't set node-label-expression to it.

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , I think I made a mistake in previous comment, rack/node local resource request is not allowed to specify label-expression for now. See SchedulerUtils.validateResourceRequest, I think you need to add a check in MR side, if the resourceName is not ANY, don't set node-label-expression to it.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda
          I have already taken care of this

          • JOB_NODE_LABEL_EXP is set to the Application submission context and yarn takes care of setting only when ResourceRequest is ANY
          • AM_NODE_LABEL_EXP is set to the AM's Resource request which is of Any type always.
          • MAP_NODE_LABEL_EXP & REDUCE_NODE_LABEL_EXP are added to the resource request only when its of ANY type in RMContainerRequestor.addContainerReq(ContainerRequest). Test case is also present for it.

          My question was in most cases container is assigned to either Node Local or Rack local, So in these cases container cannot be assigned to a Node satisfying the Node Label exp. So my suggestion was based on boolean config flag can we decide whether to add node local and rack local request or not. for example expose ENABLE_ANY_RESOURCE_REQUEST_ONLY config based on which node local and rack local request will not be set.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda I have already taken care of this JOB_NODE_LABEL_EXP is set to the Application submission context and yarn takes care of setting only when ResourceRequest is ANY AM_NODE_LABEL_EXP is set to the AM's Resource request which is of Any type always. MAP_NODE_LABEL_EXP & REDUCE_NODE_LABEL_EXP are added to the resource request only when its of ANY type in RMContainerRequestor.addContainerReq(ContainerRequest). Test case is also present for it. My question was in most cases container is assigned to either Node Local or Rack local, So in these cases container cannot be assigned to a Node satisfying the Node Label exp. So my suggestion was based on boolean config flag can we decide whether to add node local and rack local request or not. for example expose ENABLE_ANY_RESOURCE_REQUEST_ONLY config based on which node local and rack local request will not be set.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R,
          Thanks for replying, it's glad to hear such cases are already addressed.

          I think that will be problematic if we enable_any_resource_request_only, now we calculate pending-resource-by-label only for ANY request. I think it's fine to me to leave behind the node-local/rack-local out of partition issue. MR job can still get start and running.

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , Thanks for replying, it's glad to hear such cases are already addressed. I think that will be problematic if we enable_any_resource_request_only, now we calculate pending-resource-by-label only for ANY request. I think it's fine to me to leave behind the node-local/rack-local out of partition issue. MR job can still get start and running.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for replying Tan, Wangda

          Thanks for replying, it's glad to hear such cases are already addressed.

          I think that will be problematic if we enable_any_resource_request_only, now we calculate pending-resource-by-label only for ANY request. I think it's fine to me to leave behind the node-local/rack-local out of partition issue. MR job can still get start and running.

          I am ok with not adding, but my intention was not to add node-local/rack-local resource requests at all so that it there are no issues with the ones mentioned in YARN-2694. Also in small clusters(<50) all are in the same rack i.e. default rack and suppose i want to run the reducers in the nodes with good HW config (high mem/fast cpu) then this kind of config might be helpful.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for replying Tan, Wangda Thanks for replying, it's glad to hear such cases are already addressed. I think that will be problematic if we enable_any_resource_request_only, now we calculate pending-resource-by-label only for ANY request. I think it's fine to me to leave behind the node-local/rack-local out of partition issue. MR job can still get start and running. I am ok with not adding, but my intention was not to add node-local/rack-local resource requests at all so that it there are no issues with the ones mentioned in YARN-2694 . Also in small clusters(<50) all are in the same rack i.e. default rack and suppose i want to run the reducers in the nodes with good HW config (high mem/fast cpu) then this kind of config might be helpful.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda,
          Any thoughts/updates on my previous comment ? if no more changes then can we get this in ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda , Any thoughts/updates on my previous comment ? if no more changes then can we get this in ?
          Hide
          leftnoteasy Wangda Tan added a comment -

          It's the last call of comments, I plan to get this in today.

          Show
          leftnoteasy Wangda Tan added a comment - It's the last call of comments, I plan to get this in today.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7909 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7909/)
          MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7909 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7909/ ) MAPREDUCE-6304 . Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          Hide
          leftnoteasy Wangda Tan added a comment -

          Just committed to branch-2/trunk. Thanks Naganarasimha G R and review from Jian Fang and Yuliya Feldman. I cannot resolve it because JIRA system is so slow today.

          Show
          leftnoteasy Wangda Tan added a comment - Just committed to branch-2/trunk. Thanks Naganarasimha G R and review from Jian Fang and Yuliya Feldman . I cannot resolve it because JIRA system is so slow today.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for reviewing and commiting Tan, Wangda,Jian Fang & Yuliya Feldman.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for reviewing and commiting Tan, Wangda , Jian Fang & Yuliya Feldman .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/211/)
          MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/211/ ) MAPREDUCE-6304 . Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #941 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/941/)
          MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #941 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/941/ ) MAPREDUCE-6304 . Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2139/)
          MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2139/ ) MAPREDUCE-6304 . Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/199/)
          MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/199/ ) MAPREDUCE-6304 . Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/209/)
          MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/209/ ) MAPREDUCE-6304 . Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2157/)
          MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2157/ ) MAPREDUCE-6304 . Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) (wangda: rev 3164e7d83875aa6b7435d1dfe61ac280aa277f1c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R, I just found there's one findbugs warning, I missed that before committing, could you reopen this issue and add an addendum patch to fix that?

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , I just found there's one findbugs warning, I missed that before committing, could you reopen this issue and add an addendum patch to fix that? Thanks,
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R, I just found the problem is already resolved by MAPREDUCE-6421, please ignore my previous comment.

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , I just found the problem is already resolved by MAPREDUCE-6421 , please ignore my previous comment.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda
          Thanks for informing but while working on this jira had looked into it and commented also, but missed to see the jira MAPREDUCE-6421 and could have been added to the exclude list ! Will keep this in mind for next time .

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda Thanks for informing but while working on this jira had looked into it and commented also, but missed to see the jira MAPREDUCE-6421 and could have been added to the exclude list ! Will keep this in mind for next time .
          Hide
          sunilg Sunil G added a comment -

          cc/ Naganarasimha G R, Wangda Tan, Junping Du

          Currently in 2.6 branch, this patch s not present. So for users who wants to specify label at app submission time, this feature is not available . I think we can backport this to 2.6 line. I had mentioned this point in release mail thread also. Thoughts?

          Show
          sunilg Sunil G added a comment - cc/ Naganarasimha G R , Wangda Tan , Junping Du Currently in 2.6 branch, this patch s not present. So for users who wants to specify label at app submission time, this feature is not available . I think we can backport this to 2.6 line. I had mentioned this point in release mail thread also. Thoughts?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Sunil G,

          I would suggest don't backport this issue, it's a new feature instead of criticial bug fix. And I think user who wants to try node label feature should move to more stable 2.7.x release

          Show
          leftnoteasy Wangda Tan added a comment - Sunil G , I would suggest don't backport this issue, it's a new feature instead of criticial bug fix. And I think user who wants to try node label feature should move to more stable 2.7.x release
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Sunil G & Tan, Wangda,
          Though this patch can be applied with slight modification, its not a blocking bug/ feature so i am ok even if its not merged with 2.6.x as usually we try to back port only the critical/blocker issues.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Sunil G & Tan, Wangda , Though this patch can be applied with slight modification, its not a blocking bug/ feature so i am ok even if its not merged with 2.6.x as usually we try to back port only the critical/blocker issues.
          Hide
          sunilg Sunil G added a comment -

          Thanks Wangda Tan and NGarla_Unused for the clarification. It makes sense to me to backport critical/blocker alone. But one more question, since we are not porting back features, do we have a document which specifies what all functionalities of NodeLabel is available in 2.6 line. I couldn't get much information from 2.6.4 docs.

          Show
          sunilg Sunil G added a comment - Thanks Wangda Tan and NGarla_Unused for the clarification. It makes sense to me to backport critical/blocker alone. But one more question, since we are not porting back features, do we have a document which specifies what all functionalities of NodeLabel is available in 2.6 line. I couldn't get much information from 2.6.4 docs.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Sunil G,
          YARN-2801 introduced the documentation for the nodelabels and it was committed to branch 2.7.2, so even if we back port we need to ensure only the applicable part of the document needs to be backported. but the question is anyone using nodelabels in 2.6.x release ? if so taking that effort is fine else may be not worth it. Thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Sunil G , YARN-2801 introduced the documentation for the nodelabels and it was committed to branch 2.7.2, so even if we back port we need to ensure only the applicable part of the document needs to be backported. but the question is anyone using nodelabels in 2.6.x release ? if so taking that effort is fine else may be not worth it. Thoughts ?
          Hide
          sunilg Sunil G added a comment -

          Yes. I also was having similar thoughts. If we know any user s using labels in 2.6 line, we can document only that much. I have seen the doc in 307 line, and it's pretty much fine. But I agree that effort of simple backport won't be easy as we do not have all set of functionality of labels in 2.6.
          I am fine with current way, but was getting confused sometimes as Eric used a deprecated way to test labels. Also I too tested the way I do in 2.7 or trunk and met with some pblm. So it may help users if they use labels in 2.6. If its not intended, then we may not need it.

          Show
          sunilg Sunil G added a comment - Yes. I also was having similar thoughts. If we know any user s using labels in 2.6 line, we can document only that much. I have seen the doc in 307 line, and it's pretty much fine. But I agree that effort of simple backport won't be easy as we do not have all set of functionality of labels in 2.6. I am fine with current way, but was getting confused sometimes as Eric used a deprecated way to test labels. Also I too tested the way I do in 2.7 or trunk and met with some pblm. So it may help users if they use labels in 2.6. If its not intended, then we may not need it.
          Hide
          elgoiri Inigo Goiri added a comment -

          Backporting to 2.7.4

          Show
          elgoiri Inigo Goiri added a comment - Backporting to 2.7.4
          Hide
          elgoiri Inigo Goiri added a comment -

          Backport to 2.7.4.

          Show
          elgoiri Inigo Goiri added a comment - Backport to 2.7.4.
          Hide
          jhung Jonathan Hung added a comment -

          +1 (non-binding) for the 2.7.4 patch.

          Show
          jhung Jonathan Hung added a comment - +1 (non-binding) for the 2.7.4 patch.
          Hide
          shv Konstantin Shvachko added a comment -

          +1 for backport. Thanks Inigo Goiri for taking on it. Will commit shortly.

          Show
          shv Konstantin Shvachko added a comment - +1 for backport. Thanks Inigo Goiri for taking on it. Will commit shortly.
          Hide
          shv Konstantin Shvachko added a comment -

          I just committed this to branch-2.7. Thanks Inigo Goiri for backport.

          Show
          shv Konstantin Shvachko added a comment - I just committed this to branch-2.7. Thanks Inigo Goiri for backport.

            People

            • Assignee:
              Naganarasimha Naganarasimha G R
              Reporter:
              john.jian.fang Jian Fang
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development