Hadoop Common
  1. Hadoop Common
  2. HADOOP-4487 Security features for Hadoop
  3. HADOOP-4930

Implement setuid executable for Linux to assist in launching tasks as job owners

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-4490 tracks the design and implementation of launching tasks as job owners. As per discussion there, the proposal is to implement a setuid executable for Linux that would be launched by the tasktracker to run tasks as the job owners. In this task we will track the implementation of the setuid executable. HADOOP-4490 will be used to track changes in the tasktracker.

      1. HADOOP-4930-1.patch
        167 kB
        Sreekanth Ramakrishnan
      2. HADOOP-4930-10.patch
        182 kB
        Sreekanth Ramakrishnan
      3. HADOOP-4930-11.patch
        182 kB
        Sreekanth Ramakrishnan
      4. HADOOP-4930-12.patch
        184 kB
        Sreekanth Ramakrishnan
      5. HADOOP-4930-13.patch
        183 kB
        Sreekanth Ramakrishnan
      6. HADOOP-4930-2.patch
        167 kB
        Sreekanth Ramakrishnan
      7. HADOOP-4930-3.patch
        166 kB
        Sreekanth Ramakrishnan
      8. HADOOP-4930-4.patch
        177 kB
        Sreekanth Ramakrishnan
      9. HADOOP-4930-5.patch
        179 kB
        Sreekanth Ramakrishnan
      10. HADOOP-4930-6.patch
        183 kB
        Sreekanth Ramakrishnan
      11. HADOOP-4930-7.patch
        184 kB
        Sreekanth Ramakrishnan
      12. HADOOP-4930-8.patch
        184 kB
        Sreekanth Ramakrishnan
      13. HADOOP-4930-9.patch
        182 kB
        Sreekanth Ramakrishnan
      14. hadoop-4930v20.patch
        183 kB
        Robert Chansler

        Issue Links

          Activity

          Hemanth Yamijala created issue -
          Hemanth Yamijala made changes -
          Field Original Value New Value
          Parent HADOOP-4487 [ 12407018 ]
          Issue Type Task [ 3 ] Sub-task [ 7 ]
          Hemanth Yamijala made changes -
          Link This issue blocks HADOOP-4490 [ HADOOP-4490 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching patch with initial implementation of setuid script, following are the features of the patch:

          1. Spawns of task as the user who is passed to the script.
          2. kills the spawned off task as the user who is passed to the script.

          The script depends on the following configuration which is present in HADOOP_CONF_DIR/task-controller.cfg:
          hadoop.tmp.dir=hadoop temp directories which TT would be using.
          hadoop.pid.dir=directory where pid files for tt is stored.
          hadoop.indent.str=used to figure out the pid file.

          The HADOOP_CONF_DIR is passed as a pre-processor directive, using auto conf.

          -
          Steps to build the binary:

          ./configure (Requires, HADOOP_CONF_DIR)
          make (builds the, binary in build folder, deletes the output from ./configure)

          TODO:
          1. Have to integrate parent pid checking in-order to figure out if the tt is the actual process which is spawning off the setuid script.

          Show
          Sreekanth Ramakrishnan added a comment - Attaching patch with initial implementation of setuid script, following are the features of the patch: 1. Spawns of task as the user who is passed to the script. 2. kills the spawned off task as the user who is passed to the script. The script depends on the following configuration which is present in HADOOP_CONF_DIR/task-controller.cfg: hadoop.tmp.dir=hadoop temp directories which TT would be using. hadoop.pid.dir=directory where pid files for tt is stored. hadoop.indent.str=used to figure out the pid file. The HADOOP_CONF_DIR is passed as a pre-processor directive, using auto conf. - Steps to build the binary: ./configure (Requires, HADOOP_CONF_DIR) make (builds the, binary in build folder, deletes the output from ./configure) TODO: 1. Have to integrate parent pid checking in-order to figure out if the tt is the actual process which is spawning off the setuid script.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-1.patch [ 12396585 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching new patch, incorporating fix for bug found by Hemanth, allocating extra byte while doing strncpy.

          TODO's : change the configuration reading method to use fgets by allocating large buffer. make autoconfigure script accept hadoop_conf_dir as argument.

          Show
          Sreekanth Ramakrishnan added a comment - Attaching new patch, incorporating fix for bug found by Hemanth, allocating extra byte while doing strncpy. TODO's : change the configuration reading method to use fgets by allocating large buffer. make autoconfigure script accept hadoop_conf_dir as argument.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-2.patch [ 12396590 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching latest patch:

          Change log:

          • Changed autoconf script in following way:
            • --with-confdir has to be passed when running configure. The value of the argument should be the HADOOP_CONF_DIR where the binary would be deployed.
            • --prefix is used to denote the location where the binary should be installed, the default prefix has been changed to the current directory, (probably wrong idea, but this was done not to pollute the binary path in the machine where make install is run)
            • config.h has been removed and task-controller.h is the header file which is now generated by configure
          • task-controller.c: now uses fgets to read from configuration file, instead of complicated fgetc.
          Show
          Sreekanth Ramakrishnan added a comment - Attaching latest patch: Change log: Changed autoconf script in following way: --with-confdir has to be passed when running configure. The value of the argument should be the HADOOP_CONF_DIR where the binary would be deployed. --prefix is used to denote the location where the binary should be installed, the default prefix has been changed to the current directory, (probably wrong idea, but this was done not to pollute the binary path in the machine where make install is run) config.h has been removed and task-controller.h is the header file which is now generated by configure task-controller.c: now uses fgets to read from configuration file, instead of complicated fgetc.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-3.patch [ 12396657 ]
          Hide
          Hemanth Yamijala added a comment -

          Some comments:

          • Include ASF header in all files.
          • Returning different error codes might help, rather than return -1 for all types of errors.
          • Can we change the LOGFILE through an option. By default, it points to stderr
          • There is a check in get_user_details that seems redundant. Since this is a short lived executable and the user details are retrieved right at the beginning and not modified later, the check there doesn't seem necessary.
          • In main.c, commandArguments is unnecessary. Since they are read only strings, you could directly use the argv pointers.
          • mallocs are not allocating an additional byte for null-termination.
          • At a couple of places, return values for APIs and system calls are not being checked - for e.g. malloc, fopen, strtok, etc.
          • In get_configs, the temp buffer doesn't need to be malloced. It can be a fixed size buffer. In general, please review and try to reduce the number of dynamic allocations where possible.
          • Please ensure that the conf file parsing works even in face of invalid inputs in the conf file. Invalid lines should be either discarded or the exe should fail on such inputs. If we choose to exit on failures, supporting a # to indicate commented lines would be good.
          • Define constants for numeric literals
          • For the small number of config items we have, I don't think maintaining them sorted is necessary. Likewise, search can be a simple linear scan.
          • We seem to be holding only 10 values in the configuration. This is lot for now, but shouldn't this be dynamic ?
          • We don't need to malloc again for the configuration keys and values. Since the values are read only, pointers to them can be returned where needed. And this structure is not being freed at the end of execution.
          • In the function to check tt root, I think we should check for an exact match of the strings, strncmp might make substrings pass
          • For the first version, where file and directory permissions aren't being guarded, we'll need to set a umask to create files readable by all.
          • hadoop.temp.dir should be a list of paths.
          • In get_task_file_path, additional memory is being allocated for the task id.
          • Some debug prints are going to stderr, and not the LOGFILE.
          • In the case of launching JVMs, any cleanup should be done before the call to exec the script, because exec would not return if successful.
          • Please include some more documentation about the functions, and the files in general.
          • Error messages can give some more information to users about the error.
          Show
          Hemanth Yamijala added a comment - Some comments: Include ASF header in all files. Returning different error codes might help, rather than return -1 for all types of errors. Can we change the LOGFILE through an option. By default, it points to stderr There is a check in get_user_details that seems redundant. Since this is a short lived executable and the user details are retrieved right at the beginning and not modified later, the check there doesn't seem necessary. In main.c, commandArguments is unnecessary. Since they are read only strings, you could directly use the argv pointers. mallocs are not allocating an additional byte for null-termination. At a couple of places, return values for APIs and system calls are not being checked - for e.g. malloc, fopen, strtok, etc. In get_configs, the temp buffer doesn't need to be malloced. It can be a fixed size buffer. In general, please review and try to reduce the number of dynamic allocations where possible. Please ensure that the conf file parsing works even in face of invalid inputs in the conf file. Invalid lines should be either discarded or the exe should fail on such inputs. If we choose to exit on failures, supporting a # to indicate commented lines would be good. Define constants for numeric literals For the small number of config items we have, I don't think maintaining them sorted is necessary. Likewise, search can be a simple linear scan. We seem to be holding only 10 values in the configuration. This is lot for now, but shouldn't this be dynamic ? We don't need to malloc again for the configuration keys and values. Since the values are read only, pointers to them can be returned where needed. And this structure is not being freed at the end of execution. In the function to check tt root, I think we should check for an exact match of the strings, strncmp might make substrings pass For the first version, where file and directory permissions aren't being guarded, we'll need to set a umask to create files readable by all. hadoop.temp.dir should be a list of paths. In get_task_file_path, additional memory is being allocated for the task id. Some debug prints are going to stderr, and not the LOGFILE. In the case of launching JVMs, any cleanup should be done before the call to exec the script, because exec would not return if successful. Please include some more documentation about the functions, and the files in general. Error messages can give some more information to users about the error.
          Hide
          Sreekanth Ramakrishnan added a comment -

          ASF Header to all files have been included.

          A new enum which returns errors codes based on what is defined in enums.

          Now executable accepts l file name or -log=File name as input to logfile. Log file is opened in append mode with permissions of rwxrw-rw. If no log file is provided then defaults to stderr. All print statement now points to the log file.

          Removed redundant check in get_user_details.

          Function signatures have been changed to accept job id, task id and tt_root directly instead of passing an array all of the arguments are derived from argv.

          Malloc's reduced and wherever it is used an extra byte is allocated for '\0' and all strings forcibly terminated with '\0'.

          Number of dynamic memory allocation :
          get_configs - Allocating for configuration file name, contents of the file and configuration details which is right now dynamically re sized. free of contents and file name done in same function, whereas freeing up of configuration done in run_task_as_user and kill_user_task.
          get_pid_path - pid_path is malloced (freed in run_task_as_user, kill_user_task).
          get_task_script_path : task-script-path malloced but not freed, leaking this string.

          All syscalls i.e like fopen, malloc are checked for return codes. Strtok and strncat are checked for successful completion.

          config parsing fails if the file is of not the right format, the executable fails, cleaning up all memory allocated with the same. Configuration file now supports a comment line by # in begining of line, but # in middle of a line is taken as the value of the line.

          Sorting removed.

          Number of numeric literals :
          get_configs : 1. for number of bytes to read into contents (const int within the method).
          2. for number of config items max_size (const int with in the function).

          pid_file generated is now world readable and writable.

          Check tt_root now uses strcmp instead of strncmp.

          get_task_file_path is now allocating correct number of memory.

          all print statements go to LOGFILE.

          All cleaning up done before launching JVM except a string which is used for task_script_path.

          hadoop_temp_dir contains a comma sperated list of tt_roots, I have used char * as storage, do u want me to change it to char **?

          Show
          Sreekanth Ramakrishnan added a comment - ASF Header to all files have been included. A new enum which returns errors codes based on what is defined in enums. Now executable accepts l file name or -log=File name as input to logfile. Log file is opened in append mode with permissions of rwxrw-rw . If no log file is provided then defaults to stderr. All print statement now points to the log file. Removed redundant check in get_user_details. Function signatures have been changed to accept job id, task id and tt_root directly instead of passing an array all of the arguments are derived from argv. Malloc's reduced and wherever it is used an extra byte is allocated for '\0' and all strings forcibly terminated with '\0'. Number of dynamic memory allocation : get_configs - Allocating for configuration file name, contents of the file and configuration details which is right now dynamically re sized. free of contents and file name done in same function, whereas freeing up of configuration done in run_task_as_user and kill_user_task. get_pid_path - pid_path is malloced (freed in run_task_as_user, kill_user_task). get_task_script_path : task-script-path malloced but not freed, leaking this string. All syscalls i.e like fopen, malloc are checked for return codes. Strtok and strncat are checked for successful completion. config parsing fails if the file is of not the right format, the executable fails, cleaning up all memory allocated with the same. Configuration file now supports a comment line by # in begining of line, but # in middle of a line is taken as the value of the line. Sorting removed. Number of numeric literals : get_configs : 1. for number of bytes to read into contents (const int within the method). 2. for number of config items max_size (const int with in the function). pid_file generated is now world readable and writable. Check tt_root now uses strcmp instead of strncmp. get_task_file_path is now allocating correct number of memory. all print statements go to LOGFILE. All cleaning up done before launching JVM except a string which is used for task_script_path. hadoop_temp_dir contains a comma sperated list of tt_roots, I have used char * as storage, do u want me to change it to char **?
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-4.patch [ 12396730 ]
          Hide
          Hemanth Yamijala added a comment -

          This is looking good. But I do have some more comments.

          • main seems to assume the same number of arguments for all calls to the taskcontroller. While that is true for the two commands we implemented currently, it might not be true in future. It may be better to move the arguments processing per command.
          • umask needs to be set before launching the executable. This is because the child java process creates files that should have the right permissions.
          • Most of the comments that remain are related to the configuration parsing. I have two suggestions:
            • As a first option, if we can find a package that would do this parsing automatically for us, and that package is easy to include with Hadoop (or better still, already is or is widely available), then that would be great to use, and we can just do away with all the complex parsing logic.
            • If not, at a minimum, let's move all configuration related code to a separate file so that it can be managed easily.
          • Some configuration related comments:
            • It seems good to have all state of configuration in a separate struct, something like:
              struct configuration {
                int size;
                struct confentry** confdetails;
              };
              
            • get_value is returning nothing when there's an error in get_configs, where a NULL needs to be returned.
            • the buffer length passed to the fgets call should actually be decremented by the amount already read, since fgets will return on seeing a newline immediately.
            • strncat does not specifically return any error related values. The check for NULL seems unnecessary.
            • The config entries are not being freed correctly on error conditions. Basically, free_configurations should be enhanced to handle partially allocated entries.
            • The contents buffer should not be freed at the end of parsing, but just before the exec call. This is because configuration is holding pointers to the buffer as keys and values.
            • search_conf variable is unused.
            • It will be more conventional to have the length of strings being calculated to have the exact count, and add a character for null termination only in the malloc / realloc calls. This way, other calls like snprintf can use strlen directly without having to worry about the null character.
            • It is probably also safer to memset allocated buffers with 0s.
          Show
          Hemanth Yamijala added a comment - This is looking good. But I do have some more comments. main seems to assume the same number of arguments for all calls to the taskcontroller. While that is true for the two commands we implemented currently, it might not be true in future. It may be better to move the arguments processing per command. umask needs to be set before launching the executable. This is because the child java process creates files that should have the right permissions. Most of the comments that remain are related to the configuration parsing. I have two suggestions: As a first option, if we can find a package that would do this parsing automatically for us, and that package is easy to include with Hadoop (or better still, already is or is widely available), then that would be great to use, and we can just do away with all the complex parsing logic. If not, at a minimum, let's move all configuration related code to a separate file so that it can be managed easily. Some configuration related comments: It seems good to have all state of configuration in a separate struct, something like: struct configuration { int size; struct confentry** confdetails; }; get_value is returning nothing when there's an error in get_configs, where a NULL needs to be returned. the buffer length passed to the fgets call should actually be decremented by the amount already read, since fgets will return on seeing a newline immediately. strncat does not specifically return any error related values. The check for NULL seems unnecessary. The config entries are not being freed correctly on error conditions. Basically, free_configurations should be enhanced to handle partially allocated entries. The contents buffer should not be freed at the end of parsing, but just before the exec call. This is because configuration is holding pointers to the buffer as keys and values. search_conf variable is unused. It will be more conventional to have the length of strings being calculated to have the exact count, and add a character for null termination only in the malloc / realloc calls. This way, other calls like snprintf can use strlen directly without having to worry about the null character. It is probably also safer to memset allocated buffers with 0s.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching latest patch, incorporating Hemanth's comments :

          • moved assignment of tt-root within swith statements.
          • setting umask before execlp
          • Seperated out configuration management into seperate module.
          • Changed the cleanup mechanism.
          • Corrected few places where files were not closed properly.
          Show
          Sreekanth Ramakrishnan added a comment - Attaching latest patch, incorporating Hemanth's comments : moved assignment of tt-root within swith statements. setting umask before execlp Seperated out configuration management into seperate module. Changed the cleanup mechanism. Corrected few places where files were not closed properly.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-5.patch [ 12396777 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching patch incorporating Hemanth's offline comment.

          • Made changes in build.xml to introduce a new target task-controller to build task-controller binary.
            • target requires -Dhadoop.conf.dir or HADOOP_CONF_DIR set.
              • if HADOOP_CONF_DIR is set and hadoop.conf.dir is passed, then hadoop.conf.dir value is used.
              • if either one of them is defined then that value is used.
              • if none of t hem is defined then build fails.
            • Default install directory for the binary is distribution directory/bin. This can be overriden by passing -Dtask-controller.install.dir=$HADOOP_HOME/bin
          Show
          Sreekanth Ramakrishnan added a comment - Attaching patch incorporating Hemanth's offline comment. Made changes in build.xml to introduce a new target task-controller to build task-controller binary. target requires -Dhadoop.conf.dir or HADOOP_CONF_DIR set. if HADOOP_CONF_DIR is set and hadoop.conf.dir is passed, then hadoop.conf.dir value is used. if either one of them is defined then that value is used. if none of t hem is defined then build fails. Default install directory for the binary is distribution directory/bin. This can be overriden by passing -Dtask-controller.install.dir=$HADOOP_HOME/bin
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-6.patch [ 12396894 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching new patch, fixing issue pointed out by Hemanth offline.

          • Changing ownership of the created .pid file which is created to the launching user.
          • Had commented out umask setting, which is reverted in this patch.
          Show
          Sreekanth Ramakrishnan added a comment - Attaching new patch, fixing issue pointed out by Hemanth offline. Changing ownership of the created .pid file which is created to the launching user. Had commented out umask setting, which is reverted in this patch.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-7.patch [ 12396903 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching the latest patch with following changes:

          • Made changes now to accept mapred.local.dir instead of hadoop.tmp.dir
          • Made changes in build so that now it builds the binary, does not make it a setuid executable. The permission and ownership change is left out as a deployment step.
          Show
          Sreekanth Ramakrishnan added a comment - Attaching the latest patch with following changes: Made changes now to accept mapred.local.dir instead of hadoop.tmp.dir Made changes in build so that now it builds the binary, does not make it a setuid executable. The permission and ownership change is left out as a deployment step.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-8.patch [ 12397750 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching latest patch fixing bugs which arose while testing the setuid binary alongwith tasktracker changes.

          • Changed configuration reading, now using getline() instead of managing string buffers in the code.
          • Moved source directory location to src/c++/task-controller.
          • Made appropriate changes in build script to point to correct source location.

          ant test-path result are off the mark, it is showing +1 to test when no test is involved:

               [exec]
               [exec] -1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 1 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec]
               [exec]     -1 release audit.  The applied patch generated 820 release audit warnings (more than the trunk's current 818 warnings).
          
          
          Show
          Sreekanth Ramakrishnan added a comment - Attaching latest patch fixing bugs which arose while testing the setuid binary alongwith tasktracker changes. Changed configuration reading, now using getline() instead of managing string buffers in the code. Moved source directory location to src/c++/task-controller. Made appropriate changes in build script to point to correct source location. ant test-path result are off the mark, it is showing +1 to test when no test is involved: [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 1 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] -1 release audit. The applied patch generated 820 release audit warnings (more than the trunk's current 818 warnings).
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-9.patch [ 12399616 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          The release audit warnings are because I have introduced a new configuration file and having an autogenerated configure script generated by autoconf.

          Show
          Sreekanth Ramakrishnan added a comment - The release audit warnings are because I have introduced a new configuration file and having an autogenerated configure script generated by autoconf.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching latest patch with Giri's offline comments:

          • Added autoreconf target which all c++ projects use in create-c++-configure target.
          • Made autoconf version check to 2.59
          • Removed a superfluous target called checkproperty, as currently HADOOP_CONF_DIR is taken as "../conf" when task-controller is built, i.e. binary would be built with a relative path to task-controller.cfg. If users require to change it to absolute path can pass it on as -Dhadoop.conf.dir
          Show
          Sreekanth Ramakrishnan added a comment - Attaching latest patch with Giri's offline comments: Added autoreconf target which all c++ projects use in create-c++-configure target. Made autoconf version check to 2.59 Removed a superfluous target called checkproperty, as currently HADOOP_CONF_DIR is taken as "../conf" when task-controller is built, i.e. binary would be built with a relative path to task-controller.cfg. If users require to change it to absolute path can pass it on as -Dhadoop.conf.dir
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-10.patch [ 12399618 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Missed out of few comments:

          • Changing permission of the configure script which is bundled with task-controller to include execute.
          • Change the property of task-controller.src to c++.task-controller.src

          Thank Giri for reviewing, the release and build part of the patch.

          Show
          Sreekanth Ramakrishnan added a comment - Missed out of few comments: Changing permission of the configure script which is bundled with task-controller to include execute. Change the property of task-controller.src to c++.task-controller.src Thank Giri for reviewing, the release and build part of the patch.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-11.patch [ 12399622 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching latest patch incorporating following offline comments from Hemanth:

          • Changes made to build task-controller, now if no hadoop.conf.dir is passed to build, then binary derives the path of the hadoop.conf.dir. The path which binary is present is assumed to be $HADOOP_HOME/bin/task-controller and conf directory is assumed to be $HADOOP_HOME/conf. If custom conf directory has to be used, the same can be done by passing -Dhadoop.conf.dir=<directory> as option while building task-controller.
          • Cleaning up of malloced variables in configuration.c task-controller.c

          output from ant test-patch is :

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 1 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     -1 release audit.  The applied patch generated 821 release audit warnings (more than the trunk's current 819 warnings).
          

          Release audit warning are for configuration file and autogenerated autoconf file.

          Show
          Sreekanth Ramakrishnan added a comment - Attaching latest patch incorporating following offline comments from Hemanth: Changes made to build task-controller, now if no hadoop.conf.dir is passed to build, then binary derives the path of the hadoop.conf.dir. The path which binary is present is assumed to be $HADOOP_HOME/bin/task-controller and conf directory is assumed to be $HADOOP_HOME/conf. If custom conf directory has to be used, the same can be done by passing -Dhadoop.conf.dir=<directory> as option while building task-controller. Cleaning up of malloced variables in configuration.c task-controller.c output from ant test-patch is : [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 1 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] -1 release audit. The applied patch generated 821 release audit warnings (more than the trunk's current 819 warnings). Release audit warning are for configuration file and autogenerated autoconf file.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-12.patch [ 12399798 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching latest file modifying the build.xml, incorporating Hemanths offline comment.

          • Removed the script task and replaced the same with <condition> task.
          Show
          Sreekanth Ramakrishnan added a comment - Attaching latest file modifying the build.xml, incorporating Hemanths offline comment. Removed the script task and replaced the same with <condition> task.
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-4930-13.patch [ 12399811 ]
          Hide
          Hemanth Yamijala added a comment -

          Reviewed the changes. +1. I will commit this.

          Show
          Hemanth Yamijala added a comment - Reviewed the changes. +1. I will commit this.
          Hide
          Hemanth Yamijala added a comment -

          I committed this. Thanks, Sreekanth !

          Show
          Hemanth Yamijala added a comment - I committed this. Thanks, Sreekanth !
          Hemanth Yamijala made changes -
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #756 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/ )
          Hide
          Robert Chansler added a comment -

          Attached example for earlier version not to be committed.

          Show
          Robert Chansler added a comment - Attached example for earlier version not to be committed.
          Robert Chansler made changes -
          Attachment hadoop-4930v20.patch [ 12409402 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Sreekanth Ramakrishnan
              Reporter:
              Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development