Hadoop Common
  1. Hadoop Common
  2. HADOOP-466

Startup scripts will not start instances of Hadoop daemons w/different configs w/o setting separate PID directories

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.20.0
    • Component/s: conf
    • Labels:
      None

      Description

      Configuration directories can be specified by either setting HADOOP_CONF_DIR or using the --config command line option. However, the hadoop-daemon.sh script will not start the daemons unless the PID directory is separate for each configuration.

      The issue is that the code for generating PID filenames is not dependent on the configuration directory. While the PID directory can be changed in hadoop-env.sh, it seems a little unnecessary to have this restriction.

      1. hadoop-466.diff
        0.6 kB
        Vetle Roeim

        Activity

        Vetle Roeim created issue -
        Hide
        Vetle Roeim added a comment -

        This patch generates a md5 hash from the HADOOP_CONF_DIR variable, and uses that in the PID filename.

        This will fix this issue, allowing daemons to be started with separate configurations, without having to create and configure separate directories for PID files.

        Show
        Vetle Roeim added a comment - This patch generates a md5 hash from the HADOOP_CONF_DIR variable, and uses that in the PID filename. This will fix this issue, allowing daemons to be started with separate configurations, without having to create and configure separate directories for PID files.
        Vetle Roeim made changes -
        Field Original Value New Value
        Attachment hadoop-466.diff [ 12339206 ]
        Hide
        Doug Cutting added a comment -

        This looks good to me. Does anyone else have any concerns about this?

        Show
        Doug Cutting added a comment - This looks good to me. Does anyone else have any concerns about this?
        Doug Cutting made changes -
        Fix Version/s 0.6.0 [ 12312025 ]
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Yoram Arnon added a comment -

        perhaps HADOOP_IDENT_STRING is not required any more, or is indeed harmful.
        Having it enables running multiple instances on the same conf dir by running as different users.
        it may be ok for log/out files, to separate output from different runs, but for pid files it's better to use the md5 without the string

        Show
        Yoram Arnon added a comment - perhaps HADOOP_IDENT_STRING is not required any more, or is indeed harmful. Having it enables running multiple instances on the same conf dir by running as different users. it may be ok for log/out files, to separate output from different runs, but for pid files it's better to use the md5 without the string
        Hide
        Vetle Roeim added a comment -

        How does not having HADOOP_IDENT_STRING prevent running multiple instances on the same conf dir as different users? The configuration contains port numbers, and the Hadoop daemons won't start if there are already daemons bound to those ports.

        In any case, running with separate config directories has been greatly simplified with the introduction of the --config command line parameter.

        Show
        Vetle Roeim added a comment - How does not having HADOOP_IDENT_STRING prevent running multiple instances on the same conf dir as different users? The configuration contains port numbers, and the Hadoop daemons won't start if there are already daemons bound to those ports. In any case, running with separate config directories has been greatly simplified with the introduction of the --config command line parameter.
        Hide
        Yoram Arnon added a comment -

        if HADOOP_IDENT_STRING isn't specified explicitly, it defaults to $USER.
        pid files that incorporate HADOOP_IDENT_STRING will then have different names when invoked by different users, negating their effect as uniquely identifying a process using that conf dir.
        slaves bind to the configured port, but if that fails they scan forward up to 10 ports to find a port to bind, so they'll start just fine, connect the the same master and presto, we have two slaves on the same machine running as different users.

        I always use the --config parameter, so yes, I'm not overly concerned, but I've seen this happen.

        It's not a big deal, but as long as you're going to the trouble of hashing the conf dir into the file names, I thought it might be another improvement.

        Show
        Yoram Arnon added a comment - if HADOOP_IDENT_STRING isn't specified explicitly, it defaults to $USER. pid files that incorporate HADOOP_IDENT_STRING will then have different names when invoked by different users, negating their effect as uniquely identifying a process using that conf dir. slaves bind to the configured port, but if that fails they scan forward up to 10 ports to find a port to bind, so they'll start just fine, connect the the same master and presto, we have two slaves on the same machine running as different users. I always use the --config parameter, so yes, I'm not overly concerned, but I've seen this happen. It's not a big deal, but as long as you're going to the trouble of hashing the conf dir into the file names, I thought it might be another improvement.
        Hide
        Vetle Roeim added a comment -

        Ah, I didn't know it worked like this. Seems that it might be a useful feature.

        But all I want is the hashing, so does HADOOP_IDENT_STRING stay in, or should it be taken out? Perhaps this should be another issue entirely?

        Show
        Vetle Roeim added a comment - Ah, I didn't know it worked like this. Seems that it might be a useful feature. But all I want is the hashing, so does HADOOP_IDENT_STRING stay in, or should it be taken out? Perhaps this should be another issue entirely?
        Doug Cutting made changes -
        Fix Version/s 0.6.0 [ 12312025 ]
        Sameer Paranjpye made changes -
        Component/s conf [ 12310711 ]
        Doug Cutting made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Doug Cutting made changes -
        Assignee Doug Cutting [ cutting ]
        Doug Cutting made changes -
        Assignee Doug Cutting [ cutting ]
        Hide
        Harsh J added a comment -

        This problem indeed exists if one doesn't use the HADOOP_IDENT_STRING, but that's a valid workaround than adding a dependency on md5sum and the like (or do we already use it?).

        I think this may be resolved as fixed with the availability of HADOOP_IDENT_STRING to workaround with.

        Workaround (tested to work in 0.20.2):

        # To start a second DN on same machine, with separated config.
        HADOOP_IDENT_STRING=$USER-DN2 hadoop-daemon.sh --config /conf/dn2 start datanode
        HADOOP_IDENT_STRING=$USER-DN2 hadoop-daemon.sh --config /conf/dn2 stop datanode
        # These manage the PIDs as well, and will not complain that stuff is already running.
        
        Show
        Harsh J added a comment - This problem indeed exists if one doesn't use the HADOOP_IDENT_STRING, but that's a valid workaround than adding a dependency on md5sum and the like (or do we already use it?). I think this may be resolved as fixed with the availability of HADOOP_IDENT_STRING to workaround with. Workaround (tested to work in 0.20.2): # To start a second DN on same machine, with separated config. HADOOP_IDENT_STRING=$USER-DN2 hadoop-daemon.sh --config /conf/dn2 start datanode HADOOP_IDENT_STRING=$USER-DN2 hadoop-daemon.sh --config /conf/dn2 stop datanode # These manage the PIDs as well, and will not complain that stuff is already running.
        Harsh J made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.20.0 [ 12313438 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Vetle Roeim
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development