Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1207

Allow admins to set java options for map/reduce tasks

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: client, mrv2
    • Labels:
      None

      Description

      It will be useful for allow cluster-admins to set some java options for child map/reduce tasks.

      E.g. We've had to ask users to set -Djava.net.preferIPv4Stack=true in their jobs, it would be nice to do it for all users in such scenarios even when people override mapred.child.

      {map|reduce}

      .java.opts but forget to add this.

      1. mapred-1207.patch
        13 kB
        Krishna Ramachandran

        Activity

        Hide
        Todd Lipcon added a comment -

        +1 for the idea, though I wonder which should take precedence - should the admin options come first or second?

        Could this not be done by changing the default configuration to be final "-Djava.net.preferIPv4Stack=true $

        {mapred.child.java.opts.user}

        " and then having users use that property to append their options?

        Show
        Todd Lipcon added a comment - +1 for the idea, though I wonder which should take precedence - should the admin options come first or second? Could this not be done by changing the default configuration to be final "-Djava.net.preferIPv4Stack=true $ {mapred.child.java.opts.user} " and then having users use that property to append their options?
        Hide
        steve_l added a comment -

        Given IPv6 is so much trouble to Hadoop, maybe that particular setting should be everywhere anyway

        Show
        steve_l added a comment - Given IPv6 is so much trouble to Hadoop, maybe that particular setting should be everywhere anyway
        Hide
        Alejandro Abdelnur added a comment -

        A simple solution by configuration changes would be setting the following defaults:

        mapred.child.java.opts.admin.pre=-Xmx512m
        mapred.child.java.opts.user=
        mapred.child.java.opts.admin.post=
        mapred.child.java.opts=$

        {mapred.child.java.opts.admin.pre}

        $

        {mapred.child.java.opts.user}

        $

        {mapred.child.java.opts.admin.post}

        Cluster admins can set the 'admin.pre' and 'admin.post' properties to required values and make them final.

        Users can still set values using the 'user' property.

        The down side of this approach is that we are changing the current contract.

        Show
        Alejandro Abdelnur added a comment - A simple solution by configuration changes would be setting the following defaults: mapred.child.java.opts.admin.pre=-Xmx512m mapred.child.java.opts.user= mapred.child.java.opts.admin.post= mapred.child.java.opts=$ {mapred.child.java.opts.admin.pre} $ {mapred.child.java.opts.user} $ {mapred.child.java.opts.admin.post} Cluster admins can set the 'admin.pre' and 'admin.post' properties to required values and make them final. Users can still set values using the 'user' property. The down side of this approach is that we are changing the current contract.
        Hide
        Arun C Murthy added a comment -

        Forcing users to set 'mapred.child.java.opts.user' rather than mapred.child.java.opts is an incompabtible change.

        Show
        Arun C Murthy added a comment - Forcing users to set 'mapred.child.java.opts.user' rather than mapred.child.java.opts is an incompabtible change.
        Hide
        Krishna Ramachandran added a comment -

        With this patch cluster admin can configure these

        mapreduce.admin.user.shell
        mapreduce.admin.user.home.dir

        and any custom environment variable using

        mapreduce.admin.user.env

        Show
        Krishna Ramachandran added a comment - With this patch cluster admin can configure these mapreduce.admin.user.shell mapreduce.admin.user.home.dir and any custom environment variable using mapreduce.admin.user.env
        Hide
        Krishna Ramachandran added a comment -

        suggested fix

        Show
        Krishna Ramachandran added a comment - suggested fix
        Hide
        Dick King added a comment -

        TaskRunner.java :

        API nomenclature: MAPRED_MAP_ADMIN_JAVA_OPTS etc. is a property that contains options, not a set of options like DEFAULT_MAPRED_ADMIN_JAVA_OPTS , and should probably be named MAP_ADMIN_JAVA_OPTS_PROPNAME . There should probably be such a property for mapred as well as the two separate properties for map and reduce.

        funny return value: getVMEnvironment [and after the patch, {{setEnvFromInputString}} ] only ever returns its first parameter, errorInfo , if it returns at all. The return value is certainly not pulling its weight, and the method should be void .

        functionality lack: Environment value substitution is extremely specialized. It can basically only handle expansions to path-like environment variables.

        The original code had this property, but with the new code it looks like we should to be able to handle more cases. In particular, updateUserLoginEnv(...) }} sets several environment variables such as {{USER . The administrator is likely to want to be able to expand $USER in a new environment variable value [ie., perhaps a path name].

        TestTaskEnvironment.java

        Environment variable substitution is not tested, even the limited version that we support.

        Show
        Dick King added a comment - TaskRunner.java : API nomenclature: MAPRED_MAP_ADMIN_JAVA_OPTS etc. is a property that contains options, not a set of options like DEFAULT_MAPRED_ADMIN_JAVA_OPTS , and should probably be named MAP_ADMIN_JAVA_OPTS_PROPNAME . There should probably be such a property for mapred as well as the two separate properties for map and reduce. funny return value: getVMEnvironment [and after the patch, {{setEnvFromInputString}} ] only ever returns its first parameter, errorInfo , if it returns at all. The return value is certainly not pulling its weight, and the method should be void . functionality lack: Environment value substitution is extremely specialized. It can basically only handle expansions to path-like environment variables. The original code had this property, but with the new code it looks like we should to be able to handle more cases. In particular, updateUserLoginEnv(...) }} sets several environment variables such as {{USER . The administrator is likely to want to be able to expand $USER in a new environment variable value [ie., perhaps a path name] . TestTaskEnvironment.java Environment variable substitution is not tested, even the limited version that we support.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12451069/mapred-1207.patch
        against trunk revision 1075216.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/90//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/90//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/90//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451069/mapred-1207.patch against trunk revision 1075216. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/90//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/90//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/90//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.

        Given this is not an issue with MRv2 should we still commit this? I'm happy to, but not sure it's useful. Thanks.

        We should port to NodeManager...

        Show
        Arun C Murthy added a comment - Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks. Given this is not an issue with MRv2 should we still commit this? I'm happy to, but not sure it's useful. Thanks. We should port to NodeManager...
        Hide
        Arun C Murthy added a comment -

        Fixed in trunk/branch-0.23 via MAPREDUCE-2880.

        Show
        Arun C Murthy added a comment - Fixed in trunk/branch-0.23 via MAPREDUCE-2880 .

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development