Bigtop
  1. Bigtop
  2. BIGTOP-716

standardize on bash as a shell of choice

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: backlog
    • Component/s: General
    • Labels:
      None

      Description

      90% of development and testing of our scripts happens with bash, however, if we don't enforce

      #!/bin/bash

      and just specify

      #!/bin/sh

      instead on some of the systems we may end up running against #!/bin/dash and other bash alternatives. This could lead to subtle problems.

        Issue Links

          Activity

          Hide
          Sean Mackrory added a comment -

          Just pushed BIGTOP-1045, which was, in part, a dupe of this one.

          Show
          Sean Mackrory added a comment - Just pushed BIGTOP-1045 , which was, in part, a dupe of this one.
          Hide
          Sean Mackrory added a comment -

          I had forgotten that this discussion had been held on JIRA and filed a separate ticket for this and some other "standardization" issues. I've submitted a patch to BIGTOP-1045, and I plan to commit it very soon...

          Show
          Sean Mackrory added a comment - I had forgotten that this discussion had been held on JIRA and filed a separate ticket for this and some other "standardization" issues. I've submitted a patch to BIGTOP-1045 , and I plan to commit it very soon...
          Hide
          Sean Mackrory added a comment -

          I just got bitten by another more subtle way that a non-BASH default shell can get in the way: when we invoke scripts with "sh <script name>" inside other scripts. We should specify BASH in those locations too.

          Show
          Sean Mackrory added a comment - I just got bitten by another more subtle way that a non-BASH default shell can get in the way: when we invoke scripts with "sh <script name>" inside other scripts. We should specify BASH in those locations too.
          Hide
          Peter Linnell added a comment -

          +1 And I would have no objection adding it as a hard dependency for Deb/Ubuntu

          Show
          Peter Linnell added a comment - +1 And I would have no objection adding it as a hard dependency for Deb/Ubuntu
          Hide
          Sean Mackrory added a comment -

          +1!

          Show
          Sean Mackrory added a comment - +1!
          Hide
          Eli Collins added a comment -

          Yea, a blanket search and replace makes sense. Perhaps use #!/usr/bin/env bash to be more portable while you're at it.

          Not sure if we need to update the package dependency, if Debian/Ubuntu (which configure dash to be the default) still install bash by default perhaps you don't need the dependency. Worth seeing what other packages do there.

          Show
          Eli Collins added a comment - Yea, a blanket search and replace makes sense. Perhaps use #!/usr/bin/env bash to be more portable while you're at it. Not sure if we need to update the package dependency, if Debian/Ubuntu (which configure dash to be the default) still install bash by default perhaps you don't need the dependency. Worth seeing what other packages do there.
          Hide
          Anatoli Fomenko added a comment -

          +1
          With testing, of course.

          Show
          Anatoli Fomenko added a comment - +1 With testing, of course.
          Hide
          Roman Shaposhnik added a comment -
          Here's what a simple git grep '#!.*[^a]sh' uncovered:
          bigtop-packages/src/common/bigtop-jsvc/do-component-build:#!/bin/sh
          bigtop-packages/src/common/bigtop-tomcat/do-component-build:#!/bin/sh
          bigtop-packages/src/common/datafu/do-component-build:#!/bin/sh
          bigtop-packages/src/common/datafu/install_datafu.sh:#!/bin/sh
          bigtop-packages/src/common/flume/do-component-build:#!/bin/sh
          bigtop-packages/src/common/flume/install_flume.sh:#!/bin/sh
          bigtop-packages/src/common/giraph/do-component-build:#!/bin/sh
          bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh
          bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh
          bigtop-packages/src/common/hadoop/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hadoop/install_hadoop.sh:#!/bin/sh
          bigtop-packages/src/common/hbase/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh
          bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh
          bigtop-packages/src/common/hive/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh
          bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh
          bigtop-packages/src/common/hue/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hue/install_hue.sh:#!/bin/sh
          bigtop-packages/src/common/mahout/do-component-build:#!/bin/sh
          bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh
          bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/oozie.init:#! /bin/sh
          bigtop-packages/src/common/pig/do-component-build:#!/bin/sh
          bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh
          bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh
          bigtop-packages/src/common/sqoop/do-component-build:#!/bin/sh
          bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh
          bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh
          bigtop-packages/src/common/whirr/do-component-build:#!/bin/sh
          bigtop-packages/src/common/whirr/install_whirr.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/do-component-build:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:echo '#!/bin/bash' > $BUILD_DIR/bin/zkServer-initialize.sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/deb/flume/flume-node.postinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.postinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.preinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.prerm:#!/bin/sh
          bigtop-packages/src/deb/giraph/giraph.postinst:#!/bin/sh
          bigtop-packages/src/deb/giraph/giraph.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/append_licenses.sh:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-hdfs.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-hdfs.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-mapreduce.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-mapreduce.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-yarn.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-yarn.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.prerm:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.postinst:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.preinst:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.prerm:#!/bin/sh
          bigtop-packages/src/deb/hbase/install_init_scripts.sh:#!/bin/sh
          bigtop-packages/src/deb/hbase/service-postinst.tpl:#!/bin/sh
          bigtop-packages/src/deb/hbase/service-postrm.tpl:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.postinst:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.preinst:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.prerm:#!/bin/sh
          bigtop-packages/src/deb/hive/install_init_scripts.sh:#!/bin/sh
          bigtop-packages/src/deb/mahout/mahout.postinst:#!/bin/sh
          bigtop-packages/src/deb/mahout/mahout.prerm:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.postinst:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.postrm:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.preinst:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.prerm:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.postinst:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.preinst:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.prerm:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop-metastore.postinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop-metastore.preinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop.postinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop.prerm:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.postinst:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.preinst:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.prerm:#!/bin/sh
          bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/mku.sh:#!/bin/sh
          

          Should we do a blanket search-n-replace? And follow up with updating package dependencies as well?

          Show
          Roman Shaposhnik added a comment - Here's what a simple git grep '#!.*[^a]sh' uncovered: bigtop-packages/src/common/bigtop-jsvc/do-component-build:#!/bin/sh bigtop-packages/src/common/bigtop-tomcat/do-component-build:#!/bin/sh bigtop-packages/src/common/datafu/do-component-build:#!/bin/sh bigtop-packages/src/common/datafu/install_datafu.sh:#!/bin/sh bigtop-packages/src/common/flume/do-component-build:#!/bin/sh bigtop-packages/src/common/flume/install_flume.sh:#!/bin/sh bigtop-packages/src/common/giraph/do-component-build:#!/bin/sh bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh bigtop-packages/src/common/hadoop/do-component-build:#!/bin/sh bigtop-packages/src/common/hadoop/install_hadoop.sh:#!/bin/sh bigtop-packages/src/common/hbase/do-component-build:#!/bin/sh bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh bigtop-packages/src/common/hive/do-component-build:#!/bin/sh bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh bigtop-packages/src/common/hue/do-component-build:#!/bin/sh bigtop-packages/src/common/hue/install_hue.sh:#!/bin/sh bigtop-packages/src/common/mahout/do-component-build:#!/bin/sh bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh bigtop-packages/src/common/oozie/oozie.init:#! /bin/sh bigtop-packages/src/common/pig/do-component-build:#!/bin/sh bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh bigtop-packages/src/common/sqoop/do-component-build:#!/bin/sh bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh bigtop-packages/src/common/whirr/do-component-build:#!/bin/sh bigtop-packages/src/common/whirr/install_whirr.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/do-component-build:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:echo '#!/bin/bash' > $BUILD_DIR/bin/zkServer-initialize.sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/deb/flume/flume-node.postinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.postinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.preinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.prerm:#!/bin/sh bigtop-packages/src/deb/giraph/giraph.postinst:#!/bin/sh bigtop-packages/src/deb/giraph/giraph.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/append_licenses.sh:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-hdfs.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-hdfs.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-mapreduce.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-mapreduce.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-yarn.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-yarn.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.prerm:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.postinst:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.preinst:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.prerm:#!/bin/sh bigtop-packages/src/deb/hbase/install_init_scripts.sh:#!/bin/sh bigtop-packages/src/deb/hbase/service-postinst.tpl:#!/bin/sh bigtop-packages/src/deb/hbase/service-postrm.tpl:#!/bin/sh bigtop-packages/src/deb/hive/hive.postinst:#!/bin/sh bigtop-packages/src/deb/hive/hive.preinst:#!/bin/sh bigtop-packages/src/deb/hive/hive.prerm:#!/bin/sh bigtop-packages/src/deb/hive/install_init_scripts.sh:#!/bin/sh bigtop-packages/src/deb/mahout/mahout.postinst:#!/bin/sh bigtop-packages/src/deb/mahout/mahout.prerm:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.postinst:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.postrm:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.preinst:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.prerm:#!/bin/sh bigtop-packages/src/deb/pig/pig.postinst:#!/bin/sh bigtop-packages/src/deb/pig/pig.preinst:#!/bin/sh bigtop-packages/src/deb/pig/pig.prerm:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop-metastore.postinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop-metastore.preinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop.postinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop.prerm:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.postinst:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.preinst:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.prerm:#!/bin/sh bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/mku.sh:#!/bin/sh Should we do a blanket search-n-replace? And follow up with updating package dependencies as well?
          Hide
          Eli Collins added a comment -

          +1

          Show
          Eli Collins added a comment - +1

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development