Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-716

standardize on bash as a shell of choice

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: backlog
    • Component/s: general
    • Labels:
      None

      Description

      90% of development and testing of our scripts happens with bash, however, if we don't enforce

      #!/bin/bash

      and just specify

      #!/bin/sh

      instead on some of the systems we may end up running against #!/bin/dash and other bash alternatives. This could lead to subtle problems.

        Issue Links

          Activity

          Hide
          eli Eli Collins added a comment -

          +1

          Show
          eli Eli Collins added a comment - +1
          Hide
          rvs Roman Shaposhnik added a comment -
          Here's what a simple git grep '#!.*[^a]sh' uncovered:
          bigtop-packages/src/common/bigtop-jsvc/do-component-build:#!/bin/sh
          bigtop-packages/src/common/bigtop-tomcat/do-component-build:#!/bin/sh
          bigtop-packages/src/common/datafu/do-component-build:#!/bin/sh
          bigtop-packages/src/common/datafu/install_datafu.sh:#!/bin/sh
          bigtop-packages/src/common/flume/do-component-build:#!/bin/sh
          bigtop-packages/src/common/flume/install_flume.sh:#!/bin/sh
          bigtop-packages/src/common/giraph/do-component-build:#!/bin/sh
          bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh
          bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh
          bigtop-packages/src/common/hadoop/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hadoop/install_hadoop.sh:#!/bin/sh
          bigtop-packages/src/common/hbase/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh
          bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh
          bigtop-packages/src/common/hive/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh
          bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh
          bigtop-packages/src/common/hue/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hue/install_hue.sh:#!/bin/sh
          bigtop-packages/src/common/mahout/do-component-build:#!/bin/sh
          bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh
          bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/oozie.init:#! /bin/sh
          bigtop-packages/src/common/pig/do-component-build:#!/bin/sh
          bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh
          bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh
          bigtop-packages/src/common/sqoop/do-component-build:#!/bin/sh
          bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh
          bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh
          bigtop-packages/src/common/whirr/do-component-build:#!/bin/sh
          bigtop-packages/src/common/whirr/install_whirr.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/do-component-build:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:echo '#!/bin/bash' > $BUILD_DIR/bin/zkServer-initialize.sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/deb/flume/flume-node.postinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.postinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.preinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.prerm:#!/bin/sh
          bigtop-packages/src/deb/giraph/giraph.postinst:#!/bin/sh
          bigtop-packages/src/deb/giraph/giraph.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/append_licenses.sh:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-hdfs.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-hdfs.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-mapreduce.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-mapreduce.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-yarn.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-yarn.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.prerm:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.postinst:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.preinst:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.prerm:#!/bin/sh
          bigtop-packages/src/deb/hbase/install_init_scripts.sh:#!/bin/sh
          bigtop-packages/src/deb/hbase/service-postinst.tpl:#!/bin/sh
          bigtop-packages/src/deb/hbase/service-postrm.tpl:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.postinst:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.preinst:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.prerm:#!/bin/sh
          bigtop-packages/src/deb/hive/install_init_scripts.sh:#!/bin/sh
          bigtop-packages/src/deb/mahout/mahout.postinst:#!/bin/sh
          bigtop-packages/src/deb/mahout/mahout.prerm:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.postinst:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.postrm:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.preinst:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.prerm:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.postinst:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.preinst:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.prerm:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop-metastore.postinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop-metastore.preinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop.postinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop.prerm:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.postinst:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.preinst:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.prerm:#!/bin/sh
          bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/mku.sh:#!/bin/sh
          

          Should we do a blanket search-n-replace? And follow up with updating package dependencies as well?

          Show
          rvs Roman Shaposhnik added a comment - Here's what a simple git grep '#!.*[^a]sh' uncovered: bigtop-packages/src/common/bigtop-jsvc/do-component-build:#!/bin/sh bigtop-packages/src/common/bigtop-tomcat/do-component-build:#!/bin/sh bigtop-packages/src/common/datafu/do-component-build:#!/bin/sh bigtop-packages/src/common/datafu/install_datafu.sh:#!/bin/sh bigtop-packages/src/common/flume/do-component-build:#!/bin/sh bigtop-packages/src/common/flume/install_flume.sh:#!/bin/sh bigtop-packages/src/common/giraph/do-component-build:#!/bin/sh bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh bigtop-packages/src/common/hadoop/do-component-build:#!/bin/sh bigtop-packages/src/common/hadoop/install_hadoop.sh:#!/bin/sh bigtop-packages/src/common/hbase/do-component-build:#!/bin/sh bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh bigtop-packages/src/common/hive/do-component-build:#!/bin/sh bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh bigtop-packages/src/common/hue/do-component-build:#!/bin/sh bigtop-packages/src/common/hue/install_hue.sh:#!/bin/sh bigtop-packages/src/common/mahout/do-component-build:#!/bin/sh bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh bigtop-packages/src/common/oozie/oozie.init:#! /bin/sh bigtop-packages/src/common/pig/do-component-build:#!/bin/sh bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh bigtop-packages/src/common/sqoop/do-component-build:#!/bin/sh bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh bigtop-packages/src/common/whirr/do-component-build:#!/bin/sh bigtop-packages/src/common/whirr/install_whirr.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/do-component-build:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:echo '#!/bin/bash' > $BUILD_DIR/bin/zkServer-initialize.sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/deb/flume/flume-node.postinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.postinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.preinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.prerm:#!/bin/sh bigtop-packages/src/deb/giraph/giraph.postinst:#!/bin/sh bigtop-packages/src/deb/giraph/giraph.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/append_licenses.sh:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-hdfs.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-hdfs.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-mapreduce.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-mapreduce.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-yarn.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-yarn.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.prerm:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.postinst:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.preinst:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.prerm:#!/bin/sh bigtop-packages/src/deb/hbase/install_init_scripts.sh:#!/bin/sh bigtop-packages/src/deb/hbase/service-postinst.tpl:#!/bin/sh bigtop-packages/src/deb/hbase/service-postrm.tpl:#!/bin/sh bigtop-packages/src/deb/hive/hive.postinst:#!/bin/sh bigtop-packages/src/deb/hive/hive.preinst:#!/bin/sh bigtop-packages/src/deb/hive/hive.prerm:#!/bin/sh bigtop-packages/src/deb/hive/install_init_scripts.sh:#!/bin/sh bigtop-packages/src/deb/mahout/mahout.postinst:#!/bin/sh bigtop-packages/src/deb/mahout/mahout.prerm:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.postinst:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.postrm:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.preinst:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.prerm:#!/bin/sh bigtop-packages/src/deb/pig/pig.postinst:#!/bin/sh bigtop-packages/src/deb/pig/pig.preinst:#!/bin/sh bigtop-packages/src/deb/pig/pig.prerm:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop-metastore.postinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop-metastore.preinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop.postinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop.prerm:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.postinst:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.preinst:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.prerm:#!/bin/sh bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/mku.sh:#!/bin/sh Should we do a blanket search-n-replace? And follow up with updating package dependencies as well?
          Hide
          anatoli.fomenko Anatoli Fomenko added a comment -

          +1
          With testing, of course.

          Show
          anatoli.fomenko Anatoli Fomenko added a comment - +1 With testing, of course.
          Hide
          eli Eli Collins added a comment -

          Yea, a blanket search and replace makes sense. Perhaps use #!/usr/bin/env bash to be more portable while you're at it.

          Not sure if we need to update the package dependency, if Debian/Ubuntu (which configure dash to be the default) still install bash by default perhaps you don't need the dependency. Worth seeing what other packages do there.

          Show
          eli Eli Collins added a comment - Yea, a blanket search and replace makes sense. Perhaps use #!/usr/bin/env bash to be more portable while you're at it. Not sure if we need to update the package dependency, if Debian/Ubuntu (which configure dash to be the default) still install bash by default perhaps you don't need the dependency. Worth seeing what other packages do there.
          Hide
          mackrorysd Sean Mackrory added a comment -

          +1!

          Show
          mackrorysd Sean Mackrory added a comment - +1!
          Hide
          plinnell Peter Linnell added a comment -

          +1 And I would have no objection adding it as a hard dependency for Deb/Ubuntu

          Show
          plinnell Peter Linnell added a comment - +1 And I would have no objection adding it as a hard dependency for Deb/Ubuntu
          Hide
          mackrorysd Sean Mackrory added a comment -

          I just got bitten by another more subtle way that a non-BASH default shell can get in the way: when we invoke scripts with "sh <script name>" inside other scripts. We should specify BASH in those locations too.

          Show
          mackrorysd Sean Mackrory added a comment - I just got bitten by another more subtle way that a non-BASH default shell can get in the way: when we invoke scripts with "sh <script name>" inside other scripts. We should specify BASH in those locations too.
          Hide
          mackrorysd Sean Mackrory added a comment -

          I had forgotten that this discussion had been held on JIRA and filed a separate ticket for this and some other "standardization" issues. I've submitted a patch to BIGTOP-1045, and I plan to commit it very soon...

          Show
          mackrorysd Sean Mackrory added a comment - I had forgotten that this discussion had been held on JIRA and filed a separate ticket for this and some other "standardization" issues. I've submitted a patch to BIGTOP-1045 , and I plan to commit it very soon...
          Hide
          mackrorysd Sean Mackrory added a comment -

          Just pushed BIGTOP-1045, which was, in part, a dupe of this one.

          Show
          mackrorysd Sean Mackrory added a comment - Just pushed BIGTOP-1045 , which was, in part, a dupe of this one.

            People

            • Assignee:
              rvs Roman Shaposhnik
              Reporter:
              rvs Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development