Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4519

Support override of jsvc binary and log file locations when launching secure datanode.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 3.0.0
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: datanode, scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      With this improvement the following options are available in release 1.2.0 and later on 1.x release stream:
      1. jsvc location can be overridden by setting environment variable JSVC_HOME. Defaults to jsvc binary packaged within the Hadoop distro.
      2. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out.
      3. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err.

      With this improvement the following options are available in release 2.0.4 and later on 2.x release stream:
      1. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out.
      2. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err.

      For overriding jsvc location on 2.x releases, here is the release notes from HDFS-2303:
      To run secure Datanodes users must install jsvc for their platform and set JSVC_HOME to point to the location of jsvc in their environment.
      Show
      With this improvement the following options are available in release 1.2.0 and later on 1.x release stream: 1. jsvc location can be overridden by setting environment variable JSVC_HOME. Defaults to jsvc binary packaged within the Hadoop distro. 2. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out. 3. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err. With this improvement the following options are available in release 2.0.4 and later on 2.x release stream: 1. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out. 2. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err. For overriding jsvc location on 2.x releases, here is the release notes from HDFS-2303 : To run secure Datanodes users must install jsvc for their platform and set JSVC_HOME to point to the location of jsvc in their environment.
    • Target Version/s:

      Description

      Currently, builds based on branch-1 bundle a specific version of jsvc pre-built for Linux, and the startup scripts hard-code the location of the output and error files. Some deployments may prefer to upgrade to a different version of jsvc, independent of the version bundled in Hadoop, and redirect its output elsewhere.

      1. HDFS-4519.1.patch
        1 kB
        Chris Nauroth
      2. HDFS-4519-branch-1.1.patch
        3 kB
        Chris Nauroth

        Issue Links

          Activity

          Hide
          Aaron T. Myers added a comment -

          Hey Chris, is there not a similar issue in branch-2? There may very well not be - just checking.

          Show
          Aaron T. Myers added a comment - Hey Chris, is there not a similar issue in branch-2? There may very well not be - just checking.
          Hide
          Chris Nauroth added a comment -

          trunk already has the capability to set the JSVC_HOME environment variable to point at the operator's preferred version of jsvc. Part of this change will back-port support for the JSVC_HOME environment variable to branch-1. Unlike trunk, we will maintain the behavior of defaulting to Hadoop's bundled version of jsvc for backwards-compatibility if JSVC_HOME is not specified. Additionally, we will introduce JSVC_OUTFILE and JSVC_ERRFILE for redirecting the output and error streams of jsvc.

          One specific usage of this is to allow use of a new feature added in recent jsvc versions to redirect outfile and errfile to stdout and stderr via the special arguments '&1' and '&2' respectively. This feature is not supported in the current version bundled in Hadoop, and commons-daemon has stopped providing pre-built binary distributions, so we cannot upgrade the bundled version.

          Show
          Chris Nauroth added a comment - trunk already has the capability to set the JSVC_HOME environment variable to point at the operator's preferred version of jsvc. Part of this change will back-port support for the JSVC_HOME environment variable to branch-1. Unlike trunk, we will maintain the behavior of defaulting to Hadoop's bundled version of jsvc for backwards-compatibility if JSVC_HOME is not specified. Additionally, we will introduce JSVC_OUTFILE and JSVC_ERRFILE for redirecting the output and error streams of jsvc. One specific usage of this is to allow use of a new feature added in recent jsvc versions to redirect outfile and errfile to stdout and stderr via the special arguments '&1' and '&2' respectively. This feature is not supported in the current version bundled in Hadoop, and commons-daemon has stopped providing pre-built binary distributions, so we cannot upgrade the bundled version.
          Hide
          Chris Nauroth added a comment -

          The attached patch introduces JSVC_HOME, JSVC_OUTFILE, and JSVC_ERRFILE environment variables in the hadoop script on branch-1. I've successfully tested this change manually on a secure cluster. I tested overriding to a manually built commons-daemon-1.0.13, and I also tested that the default behavior of using the bundled version of jsvc is still intact.

          Here is an example of launching a secure data node using this feature:

          JSVC_HOME=/home/cnauroth/commons-daemon-1.0.13-src/src/native/unix \
            HADOOP_CLASSPATH=/home/cnauroth/commons-daemon-1.0.13-src/dist/commons-daemon-1.0.13.jar \
            HADOOP_USER_CLASSPATH_FIRST=true \
            JSVC_OUTFILE='&1' \
            JSVC_ERRFILE='&2' \
            HADOOP_SECURE_DN_USER=cnauroth \
            sudo -E bin/hadoop datanode
          
          Show
          Chris Nauroth added a comment - The attached patch introduces JSVC_HOME, JSVC_OUTFILE, and JSVC_ERRFILE environment variables in the hadoop script on branch-1. I've successfully tested this change manually on a secure cluster. I tested overriding to a manually built commons-daemon-1.0.13, and I also tested that the default behavior of using the bundled version of jsvc is still intact. Here is an example of launching a secure data node using this feature: JSVC_HOME=/home/cnauroth/commons-daemon-1.0.13-src/src/ native /unix \ HADOOP_CLASSPATH=/home/cnauroth/commons-daemon-1.0.13-src/dist/commons-daemon-1.0.13.jar \ HADOOP_USER_CLASSPATH_FIRST= true \ JSVC_OUTFILE='&1' \ JSVC_ERRFILE='&2' \ HADOOP_SECURE_DN_USER=cnauroth \ sudo -E bin/hadoop datanode
          Hide
          Chris Nauroth added a comment -

          Hi, Aaron.

          I don't see an exact match for this issue, based on the comments I gave above. I see HDFS-4497, which upgrades the commons-daemon version dependency on the Java side to resolve a build issue. I've been watching and code reviewing that one. I also found HDFS-2350, an old jira you filed about how secure data nodes don't print to console. I reassigned HDFS-2350 to myself, because the patch I posted here will give us the capability to send jsvc outfile/errfile to stdout/stderr.

          If you see an existing issue that I missed though, please let me know. Thanks!

          Show
          Chris Nauroth added a comment - Hi, Aaron. I don't see an exact match for this issue, based on the comments I gave above. I see HDFS-4497 , which upgrades the commons-daemon version dependency on the Java side to resolve a build issue. I've been watching and code reviewing that one. I also found HDFS-2350 , an old jira you filed about how secure data nodes don't print to console. I reassigned HDFS-2350 to myself, because the patch I posted here will give us the capability to send jsvc outfile/errfile to stdout/stderr. If you see an existing issue that I missed though, please let me know. Thanks!
          Hide
          Chris Nauroth added a comment -

          Relating to HDFS-2303, which shows the patch that introduced JSVC_HOME on trunk.

          Show
          Chris Nauroth added a comment - Relating to HDFS-2303 , which shows the patch that introduced JSVC_HOME on trunk.
          Hide
          Jos Backus added a comment -

          While this is an interesting intermediate solution (I suggested something similar to HWX when I tried adding daemontools support to the secure datanode), wouldn't it make more sense to retire the use of jsvc altogether? This would also allow the exception in the hadoop startup wrapper to support datanode secure mode to be removed.

          Show
          Jos Backus added a comment - While this is an interesting intermediate solution (I suggested something similar to HWX when I tried adding daemontools support to the secure datanode), wouldn't it make more sense to retire the use of jsvc altogether? This would also allow the exception in the hadoop startup wrapper to support datanode secure mode to be removed.
          Hide
          Chris Nauroth added a comment -

          Hi, Jos. I'd like to suggest that we keep the scope of this jira limited to overriding the jsvc binary, the outfile, and the errfile. Since jsvc is the expected way to launch a secure data node right now, a change that retires it may have to be flagged incompatible. The patch I attached here is fairly small and fully backwards-compatible with the existing behavior.

          I'd definitely be curious to see what you have in mind though if you want to file another jira. Are you thinking that we should drop the jsvc dependency from Hadoop and push the process management responsibility to the operator, who can choose to use jsvc or daemontools or whatever other tool is preferred by that operator?

          Show
          Chris Nauroth added a comment - Hi, Jos. I'd like to suggest that we keep the scope of this jira limited to overriding the jsvc binary, the outfile, and the errfile. Since jsvc is the expected way to launch a secure data node right now, a change that retires it may have to be flagged incompatible. The patch I attached here is fairly small and fully backwards-compatible with the existing behavior. I'd definitely be curious to see what you have in mind though if you want to file another jira. Are you thinking that we should drop the jsvc dependency from Hadoop and push the process management responsibility to the operator, who can choose to use jsvc or daemontools or whatever other tool is preferred by that operator?
          Hide
          Jos Backus added a comment -

          Sorry, Chris, didn't mean to derail this ticket. I like what you propose regarding fixing/extending the jsvc support, so +1 from me

          I will file a different ticket for my enhancement suggestion (just like I did internally when the issue came up). Briefly, my suggestion was to merge the jsvc-run class code into the main datanode class, and use JNI to call setuid() to drop privileges after allocating the privilege-requiring resources (see http://www2.sys-con.com/itsg/virtualcd/java/archives/0510/Silverman/index.html for details how this can be done).

          Show
          Jos Backus added a comment - Sorry, Chris, didn't mean to derail this ticket. I like what you propose regarding fixing/extending the jsvc support, so +1 from me I will file a different ticket for my enhancement suggestion (just like I did internally when the issue came up). Briefly, my suggestion was to merge the jsvc-run class code into the main datanode class, and use JNI to call setuid() to drop privileges after allocating the privilege-requiring resources (see http://www2.sys-con.com/itsg/virtualcd/java/archives/0510/Silverman/index.html for details how this can be done).
          Hide
          Chris Nauroth added a comment -

          Also attaching trunk patch for feature parity. trunk already has JSVC_HOME, so I just needed to add JSVC_OUTFILE and JSVC_ERRFILE.

          Show
          Chris Nauroth added a comment - Also attaching trunk patch for feature parity. trunk already has JSVC_HOME, so I just needed to add JSVC_OUTFILE and JSVC_ERRFILE.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571253/HDFS-4519.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4014//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4014//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571253/HDFS-4519.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4014//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4014//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          Jenkins gave -1 for no new tests, because this patch changes only shell scripts. I have manually tested both the trunk and branch-1 patches.

          Show
          Chris Nauroth added a comment - Jenkins gave -1 for no new tests, because this patch changes only shell scripts. I have manually tested both the trunk and branch-1 patches.
          Hide
          Suresh Srinivas added a comment -

          +1 for the trunk patch.

          Show
          Suresh Srinivas added a comment - +1 for the trunk patch.
          Hide
          Suresh Srinivas added a comment -

          +1 for the branch-1 patch.

          Show
          Suresh Srinivas added a comment - +1 for the branch-1 patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3419 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3419/)
          HDFS-4519. Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3419 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3419/ ) HDFS-4519 . Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to trunk, branch-2 and branch-1. Thank you Chris!

          Show
          Suresh Srinivas added a comment - I committed the patch to trunk, branch-2 and branch-1. Thank you Chris!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #147 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/147/)
          HDFS-4519. Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #147 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/147/ ) HDFS-4519 . Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1336 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1336/)
          HDFS-4519. Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1336 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1336/ ) HDFS-4519 . Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1364 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1364/)
          HDFS-4519. Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1364 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1364/ ) HDFS-4519 . Support overriding jsvc binary and log file locations when launching secure datanode. Contributed by Chris Nauroth. (Revision 1453050) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453050 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

            People

            • Assignee:
              Chris Nauroth
              Reporter:
              Chris Nauroth
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development