Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7755

httpfs shell code has hardcoded path to bash

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
      None

      Description

      Most of shell scripts use shebang ling in the following format:
      #!/usr/bin/env bash

      But some scripts contain hardcoded "/bin/bash" which is not portable.

      Please use #!/usr/bin/env bash instead for portability.

      PS: it would be much better to switch to standard Bourne Shell /bin/sh, do these scripts really need bash?

      1. bash.patch
        1.0 kB
        Dmitry Sivachenko

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/)
          HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/ ) HDFS-7755 . httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cmccabe Colin P. McCabe added a comment -

          What system is this [that lacked /usr/bin/env]? Just wonder.

          It's been a long time! It was an embedded system I was logged into, and I think the setup was a little weird.

          I wasn't actually seriously suggesting this as a problem

          Show
          cmccabe Colin P. McCabe added a comment - What system is this [that lacked /usr/bin/env] ? Just wonder. It's been a long time! It was an embedded system I was logged into, and I think the setup was a little weird. I wasn't actually seriously suggesting this as a problem
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2051 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2051/)
          HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2051 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2051/ ) HDFS-7755 . httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #101 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/101/)
          HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #101 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/101/ ) HDFS-7755 . httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2032 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2032/)
          HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2032 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2032/ ) HDFS-7755 . httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #834 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/834/)
          HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #834 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/834/ ) HDFS-7755 . httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #100 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/100/)
          HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #100 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/100/ ) HDFS-7755 . httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          trtrmitya Dmitry Sivachenko added a comment -

          What system is this? Just wonder.

          Show
          trtrmitya Dmitry Sivachenko added a comment - What system is this? Just wonder.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, guys. This looks good.

          Note that I once found a system where env was not at /usr/bin/env. I expect that someone will file a JIRA if and when they ever hit that though!

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, guys. This looks good. Note that I once found a system where env was not at /usr/bin/env . I expect that someone will file a JIRA if and when they ever hit that though!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7053 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7053/)
          HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7053 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7053/ ) HDFS-7755 . httpfs shell code has hardcoded path to bash (Dmitry Sivachenko via aw) (aw: rev 7d73202734e79beaa2db34d6b811beba7b34ee87) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          Hide
          aw Allen Wittenauer added a comment -

          +1 committed to trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - +1 committed to trunk. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12654572/bash.patch
          against trunk revision 1382ae5.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs-httpfs.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5632//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5632//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654572/bash.patch against trunk revision 1382ae5. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs-httpfs. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5632//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5632//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12654572/bash.patch
          against trunk revision 400e1bb.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs-httpfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4821//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4821//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654572/bash.patch against trunk revision 400e1bb. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs-httpfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4821//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4821//console This message is automatically generated.
          Hide
          pfg Pedro Giffuni added a comment -

          I wanted to clarify that the issue here is not *BSD (or Solaris, which is supposed to be the best platform for Java development) but the unnecessary dependency on a copyleft tool:
          http://www.apache.org/legal/resolved.html#category-x

          In light of the "ShellShock" vulnerability this discussion takes a new twist, though. It is not clear if this is the only vulnerability that will be found so it would be better to not depend on a specific shell.

          Checkbashisms is your friend:
          http://sourceforge.net/projects/checkbaskisms/

          Show
          pfg Pedro Giffuni added a comment - I wanted to clarify that the issue here is not *BSD (or Solaris, which is supposed to be the best platform for Java development) but the unnecessary dependency on a copyleft tool: http://www.apache.org/legal/resolved.html#category-x In light of the "ShellShock" vulnerability this discussion takes a new twist, though. It is not clear if this is the only vulnerability that will be found so it would be better to not depend on a specific shell. Checkbashisms is your friend: http://sourceforge.net/projects/checkbaskisms/
          Hide
          cmccabe Colin P. McCabe added a comment -

          There's a pretty large number of scripts relying on '#!/bin/bash'... I don't think the patch posted here fixes them all.

          cmccabe@keter:~/hadoopST/trunk> grep -rI '#!/bin/bash' *
          dev-support/findHangingTest.sh:#!/bin/bash
          dev-support/create-release.sh:#!/bin/bash
          hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh:#!/bin/bash
          hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh:#!/bin/bash
          hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh:#!/bin/bash
          hadoop-common-project/hadoop-kms/target/hadoop-kms-3.0.0-SNAPSHOT/libexec/kms-config.sh:#!/bin/bash
          hadoop-common-project/hadoop-kms/target/hadoop-kms-3.0.0-SNAPSHOT/sbin/kms.sh:#!/bin/bash
          hadoop-common-project/hadoop-kms/target/hadoop-kms-3.0.0-SNAPSHOT/etc/hadoop/kms-env.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/libexec/kms-config.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/libexec/httpfs-config.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/rumen2sls.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/slsrun.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/sbin/httpfs.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/sbin/kms.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/httpfs-env.sh:#!/bin/bash
          hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/kms-env.sh:#!/bin/bash
          hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh:#!/bin/bash
          hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh:#!/bin/bash
          hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh:#!/bin/bash
          hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-3.0.0-SNAPSHOT/libexec/httpfs-config.sh:#!/bin/bash
          hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-3.0.0-SNAPSHOT/sbin/httpfs.sh:#!/bin/bash
          hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-3.0.0-SNAPSHOT/etc/hadoop/httpfs-env.sh:#!/bin/bash
          hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/rumen2sls.sh:#!/bin/bash
          hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/slsrun.sh:#!/bin/bash
          hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh:#!/bin/bash
          hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh:#!/bin/bash
          hadoop-tools/hadoop-sls/target/hadoop-sls-3.0.0-SNAPSHOT/sls/bin/rumen2sls.sh:#!/bin/bash
          hadoop-tools/hadoop-sls/target/hadoop-sls-3.0.0-SNAPSHOT/sls/bin/slsrun.sh:#!/bin/bash
          hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java:      writer.println("#!/bin/bash\n\n");
          hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java:      fileWriter.write("#!/bin/bash\n\n");
          hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c:  if (fprintf(script, "#!/bin/bash\n"
          hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:      line("#!/bin/bash");
          hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java:      pout.println("#!/bin/bash");
          

          Okay, I see. The point to use /bin/sh is that most of these shell scripts are very simple and do not require any bash-specific things. And since there are systems that do not ship bash by default, this would eliminate one extra dependency. But it is not a big deal if you prefer to stick with more heavyweight bash instead.

          I don't have any objections to switching to /bin/sh, but I think that you'll find it a very challenging task. If your goal is just to get stuff working on FreeBSD, you're probably better off spending your effort elsewhere and coming back to this later. You would also need to have a vote on the main mailing lists to get a policy enacted to only use /bin/sh in the future, or else your work would quickly be undone by people adding new scripts. Again, I would support this, but it seems like the effort to reward ratio is pretty low.

          Show
          cmccabe Colin P. McCabe added a comment - There's a pretty large number of scripts relying on '#!/bin/bash'... I don't think the patch posted here fixes them all. cmccabe@keter:~/hadoopST/trunk> grep -rI '#!/bin/bash' * dev-support/findHangingTest.sh:#!/bin/bash dev-support/create-release.sh:#!/bin/bash hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh:#!/bin/bash hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh:#!/bin/bash hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh:#!/bin/bash hadoop-common-project/hadoop-kms/target/hadoop-kms-3.0.0-SNAPSHOT/libexec/kms-config.sh:#!/bin/bash hadoop-common-project/hadoop-kms/target/hadoop-kms-3.0.0-SNAPSHOT/sbin/kms.sh:#!/bin/bash hadoop-common-project/hadoop-kms/target/hadoop-kms-3.0.0-SNAPSHOT/etc/hadoop/kms-env.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/libexec/kms-config.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/libexec/httpfs-config.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/rumen2sls.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/slsrun.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/sbin/httpfs.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/sbin/kms.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/httpfs-env.sh:#!/bin/bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/kms-env.sh:#!/bin/bash hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh:#!/bin/bash hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh:#!/bin/bash hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh:#!/bin/bash hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-3.0.0-SNAPSHOT/libexec/httpfs-config.sh:#!/bin/bash hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-3.0.0-SNAPSHOT/sbin/httpfs.sh:#!/bin/bash hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-3.0.0-SNAPSHOT/etc/hadoop/httpfs-env.sh:#!/bin/bash hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/rumen2sls.sh:#!/bin/bash hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-3.0.0-SNAPSHOT/share/hadoop/tools/sls/bin/slsrun.sh:#!/bin/bash hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh:#!/bin/bash hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh:#!/bin/bash hadoop-tools/hadoop-sls/target/hadoop-sls-3.0.0-SNAPSHOT/sls/bin/rumen2sls.sh:#!/bin/bash hadoop-tools/hadoop-sls/target/hadoop-sls-3.0.0-SNAPSHOT/sls/bin/slsrun.sh:#!/bin/bash hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java: writer.println( "#!/bin/bash\n\n" ); hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java: fileWriter.write( "#!/bin/bash\n\n" ); hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/ native /container-executor/test/test-container-executor.c: if (fprintf(script, "#!/bin/bash\n" hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java: line( "#!/bin/bash" ); hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java: pout.println( "#!/bin/bash" ); Okay, I see. The point to use /bin/sh is that most of these shell scripts are very simple and do not require any bash-specific things. And since there are systems that do not ship bash by default, this would eliminate one extra dependency. But it is not a big deal if you prefer to stick with more heavyweight bash instead. I don't have any objections to switching to /bin/sh, but I think that you'll find it a very challenging task. If your goal is just to get stuff working on FreeBSD, you're probably better off spending your effort elsewhere and coming back to this later. You would also need to have a vote on the main mailing lists to get a policy enacted to only use /bin/sh in the future, or else your work would quickly be undone by people adding new scripts. Again, I would support this, but it seems like the effort to reward ratio is pretty low.
          Hide
          pfg Pedro Giffuni added a comment -

          s/non interactive strings/non-interactive scripts/

          Show
          pfg Pedro Giffuni added a comment - s/non interactive strings/non-interactive scripts/
          Hide
          pfg Pedro Giffuni added a comment -

          It is indeed better to use standard /bin/sh for non interactive strings and most linux distributions are following that policy:

          https://wiki.ubuntu.com/DashAsBinSh

          "We recommend that developers of shell scripts adhere to the POSIX standard, omitting those items flagged as XSI extensions. (This subset will be referred to as "POSIX shell" throughout the remainder of this document.) Doing so will improve portability to a variety of Unix systems, and will provide assurance that problems you encounter will be treated as bugs rather than as undocumented features! A special exception is that echo -n is guaranteed to be supported, although not other echo options (see below)."

          FWIW, we did this for Apache OpenOffice and it was very useful to have the checkbashims script (from sourceforge).

          Show
          pfg Pedro Giffuni added a comment - It is indeed better to use standard /bin/sh for non interactive strings and most linux distributions are following that policy: https://wiki.ubuntu.com/DashAsBinSh "We recommend that developers of shell scripts adhere to the POSIX standard, omitting those items flagged as XSI extensions. (This subset will be referred to as "POSIX shell" throughout the remainder of this document.) Doing so will improve portability to a variety of Unix systems, and will provide assurance that problems you encounter will be treated as bugs rather than as undocumented features! A special exception is that echo -n is guaranteed to be supported, although not other echo options (see below)." FWIW, we did this for Apache OpenOffice and it was very useful to have the checkbashims script (from sourceforge).
          Hide
          trtrmitya Dmitry Sivachenko added a comment -

          Okay, I see. The point to use /bin/sh is that most of these shell scripts are very simple and do not require any bash-specific things. And since there are systems that do not ship bash by default, this would eliminate one extra dependency. But it is not a big deal if you prefer to stick with more heavyweight bash instead.

          Show
          trtrmitya Dmitry Sivachenko added a comment - Okay, I see. The point to use /bin/sh is that most of these shell scripts are very simple and do not require any bash-specific things. And since there are systems that do not ship bash by default, this would eliminate one extra dependency. But it is not a big deal if you prefer to stick with more heavyweight bash instead.
          Hide
          gkesavan Giridharan Kesavan added a comment -

          I was against using /bin/sh and not against /usr/bin/env bash

          Show
          gkesavan Giridharan Kesavan added a comment - I was against using /bin/sh and not against /usr/bin/env bash
          Show
          trtrmitya Dmitry Sivachenko added a comment - Consider this article: http://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html
          Hide
          trtrmitya Dmitry Sivachenko added a comment -

          Debian is not the only system around.
          /bin/sh is a standard location for bourne shell.

          FreeBSD, for instance, installs bash as "/usr/local/bin/bash". That is why I propose to either use "/bin/sh" and do not rely on bash extensions in scripts, use good old Bourne shell syntax, or use "/usr/bin/env bash", which will invoke bash in a portable way and work on both debian and FreeBSD.

          Show
          trtrmitya Dmitry Sivachenko added a comment - Debian is not the only system around. /bin/sh is a standard location for bourne shell. FreeBSD, for instance, installs bash as "/usr/local/bin/bash". That is why I propose to either use "/bin/sh" and do not rely on bash extensions in scripts, use good old Bourne shell syntax, or use "/usr/bin/env bash", which will invoke bash in a portable way and work on both debian and FreeBSD.
          Hide
          gkesavan Giridharan Kesavan added a comment -

          /bin/sh is symlined to /bin/dash on debian platforms.
          So I think it makes sense to use /bin/bash and not /bin/sh.

          Show
          gkesavan Giridharan Kesavan added a comment - /bin/sh is symlined to /bin/dash on debian platforms. So I think it makes sense to use /bin/bash and not /bin/sh.
          Hide
          trtrmitya Dmitry Sivachenko added a comment -

          I am attaching a patch (against trunk)

          Show
          trtrmitya Dmitry Sivachenko added a comment - I am attaching a patch (against trunk)

            People

            • Assignee:
              trtrmitya Dmitry Sivachenko
              Reporter:
              trtrmitya Dmitry Sivachenko
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development