Bigtop
  1. Bigtop
  2. BIGTOP-716

standardize on bash as a shell of choice

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: backlog
    • Component/s: general
    • Labels:
      None

      Description

      90% of development and testing of our scripts happens with bash, however, if we don't enforce

      #!/bin/bash

      and just specify

      #!/bin/sh

      instead on some of the systems we may end up running against #!/bin/dash and other bash alternatives. This could lead to subtle problems.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          330d 20h 30m 1 Sean Mackrory 23/Aug/13 18:59
          Sean Mackrory made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Sean Mackrory added a comment -

          Just pushed BIGTOP-1045, which was, in part, a dupe of this one.

          Show
          Sean Mackrory added a comment - Just pushed BIGTOP-1045 , which was, in part, a dupe of this one.
          Hide
          Sean Mackrory added a comment -

          I had forgotten that this discussion had been held on JIRA and filed a separate ticket for this and some other "standardization" issues. I've submitted a patch to BIGTOP-1045, and I plan to commit it very soon...

          Show
          Sean Mackrory added a comment - I had forgotten that this discussion had been held on JIRA and filed a separate ticket for this and some other "standardization" issues. I've submitted a patch to BIGTOP-1045 , and I plan to commit it very soon...
          Sean Mackrory made changes -
          Link This issue is duplicated by BIGTOP-1045 [ BIGTOP-1045 ]
          Roman Shaposhnik made changes -
          Priority Minor [ 4 ] Blocker [ 1 ]
          Roman Shaposhnik made changes -
          Fix Version/s backlog [ 12324373 ]
          Fix Version/s 0.6.0 [ 12323895 ]
          Roman Shaposhnik made changes -
          Fix Version/s 0.6.0 [ 12323895 ]
          Fix Version/s 0.5.0 [ 12321865 ]
          Hide
          Sean Mackrory added a comment -

          I just got bitten by another more subtle way that a non-BASH default shell can get in the way: when we invoke scripts with "sh <script name>" inside other scripts. We should specify BASH in those locations too.

          Show
          Sean Mackrory added a comment - I just got bitten by another more subtle way that a non-BASH default shell can get in the way: when we invoke scripts with "sh <script name>" inside other scripts. We should specify BASH in those locations too.
          Hide
          Peter Linnell added a comment -

          +1 And I would have no objection adding it as a hard dependency for Deb/Ubuntu

          Show
          Peter Linnell added a comment - +1 And I would have no objection adding it as a hard dependency for Deb/Ubuntu
          Hide
          Sean Mackrory added a comment -

          +1!

          Show
          Sean Mackrory added a comment - +1!
          Hide
          Eli Collins added a comment -

          Yea, a blanket search and replace makes sense. Perhaps use #!/usr/bin/env bash to be more portable while you're at it.

          Not sure if we need to update the package dependency, if Debian/Ubuntu (which configure dash to be the default) still install bash by default perhaps you don't need the dependency. Worth seeing what other packages do there.

          Show
          Eli Collins added a comment - Yea, a blanket search and replace makes sense. Perhaps use #!/usr/bin/env bash to be more portable while you're at it. Not sure if we need to update the package dependency, if Debian/Ubuntu (which configure dash to be the default) still install bash by default perhaps you don't need the dependency. Worth seeing what other packages do there.
          Hide
          Anatoli Fomenko added a comment -

          +1
          With testing, of course.

          Show
          Anatoli Fomenko added a comment - +1 With testing, of course.
          Hide
          Roman Shaposhnik added a comment -
          Here's what a simple git grep '#!.*[^a]sh' uncovered:
          bigtop-packages/src/common/bigtop-jsvc/do-component-build:#!/bin/sh
          bigtop-packages/src/common/bigtop-tomcat/do-component-build:#!/bin/sh
          bigtop-packages/src/common/datafu/do-component-build:#!/bin/sh
          bigtop-packages/src/common/datafu/install_datafu.sh:#!/bin/sh
          bigtop-packages/src/common/flume/do-component-build:#!/bin/sh
          bigtop-packages/src/common/flume/install_flume.sh:#!/bin/sh
          bigtop-packages/src/common/giraph/do-component-build:#!/bin/sh
          bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh
          bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh
          bigtop-packages/src/common/hadoop/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hadoop/install_hadoop.sh:#!/bin/sh
          bigtop-packages/src/common/hbase/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh
          bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh
          bigtop-packages/src/common/hive/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh
          bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh
          bigtop-packages/src/common/hue/do-component-build:#!/bin/sh
          bigtop-packages/src/common/hue/install_hue.sh:#!/bin/sh
          bigtop-packages/src/common/mahout/do-component-build:#!/bin/sh
          bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh
          bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh
          bigtop-packages/src/common/oozie/oozie.init:#! /bin/sh
          bigtop-packages/src/common/pig/do-component-build:#!/bin/sh
          bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh
          bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh
          bigtop-packages/src/common/sqoop/do-component-build:#!/bin/sh
          bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh
          bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh
          bigtop-packages/src/common/whirr/do-component-build:#!/bin/sh
          bigtop-packages/src/common/whirr/install_whirr.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/do-component-build:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:echo '#!/bin/bash' > $BUILD_DIR/bin/zkServer-initialize.sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh
          bigtop-packages/src/deb/flume/flume-node.postinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.postinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.preinst:#!/bin/sh
          bigtop-packages/src/deb/flume/flume.prerm:#!/bin/sh
          bigtop-packages/src/deb/giraph/giraph.postinst:#!/bin/sh
          bigtop-packages/src/deb/giraph/giraph.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/append_licenses.sh:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-hdfs.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-hdfs.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-httpfs.prerm:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-mapreduce.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-mapreduce.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-yarn.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop-yarn.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.postinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.preinst:#!/bin/sh
          bigtop-packages/src/deb/hadoop/hadoop.prerm:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.postinst:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.preinst:#!/bin/sh
          bigtop-packages/src/deb/hbase/hbase.prerm:#!/bin/sh
          bigtop-packages/src/deb/hbase/install_init_scripts.sh:#!/bin/sh
          bigtop-packages/src/deb/hbase/service-postinst.tpl:#!/bin/sh
          bigtop-packages/src/deb/hbase/service-postrm.tpl:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.postinst:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.preinst:#!/bin/sh
          bigtop-packages/src/deb/hive/hive.prerm:#!/bin/sh
          bigtop-packages/src/deb/hive/install_init_scripts.sh:#!/bin/sh
          bigtop-packages/src/deb/mahout/mahout.postinst:#!/bin/sh
          bigtop-packages/src/deb/mahout/mahout.prerm:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.postinst:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.postrm:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.preinst:#!/bin/sh
          bigtop-packages/src/deb/oozie/oozie.prerm:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.postinst:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.preinst:#!/bin/sh
          bigtop-packages/src/deb/pig/pig.prerm:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop-metastore.postinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop-metastore.preinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop.postinst:#!/bin/sh
          bigtop-packages/src/deb/sqoop/sqoop.prerm:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.postinst:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.preinst:#!/bin/sh
          bigtop-packages/src/deb/zookeeper/zookeeper.prerm:#!/bin/sh
          bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/mku.sh:#!/bin/sh
          

          Should we do a blanket search-n-replace? And follow up with updating package dependencies as well?

          Show
          Roman Shaposhnik added a comment - Here's what a simple git grep '#!.*[^a]sh' uncovered: bigtop-packages/src/common/bigtop-jsvc/do-component-build:#!/bin/sh bigtop-packages/src/common/bigtop-tomcat/do-component-build:#!/bin/sh bigtop-packages/src/common/datafu/do-component-build:#!/bin/sh bigtop-packages/src/common/datafu/install_datafu.sh:#!/bin/sh bigtop-packages/src/common/flume/do-component-build:#!/bin/sh bigtop-packages/src/common/flume/install_flume.sh:#!/bin/sh bigtop-packages/src/common/giraph/do-component-build:#!/bin/sh bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh bigtop-packages/src/common/giraph/install_giraph.sh:#!/bin/sh bigtop-packages/src/common/hadoop/do-component-build:#!/bin/sh bigtop-packages/src/common/hadoop/install_hadoop.sh:#!/bin/sh bigtop-packages/src/common/hbase/do-component-build:#!/bin/sh bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh bigtop-packages/src/common/hbase/install_hbase.sh:#!/bin/sh bigtop-packages/src/common/hive/do-component-build:#!/bin/sh bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh bigtop-packages/src/common/hive/install_hive.sh:#!/bin/sh bigtop-packages/src/common/hue/do-component-build:#!/bin/sh bigtop-packages/src/common/hue/install_hue.sh:#!/bin/sh bigtop-packages/src/common/mahout/do-component-build:#!/bin/sh bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh bigtop-packages/src/common/mahout/install_mahout.sh:#!/bin/sh bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh bigtop-packages/src/common/oozie/install_oozie.sh:#!/bin/sh bigtop-packages/src/common/oozie/oozie.init:#! /bin/sh bigtop-packages/src/common/pig/do-component-build:#!/bin/sh bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh bigtop-packages/src/common/pig/install_pig.sh:#!/bin/sh bigtop-packages/src/common/sqoop/do-component-build:#!/bin/sh bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh bigtop-packages/src/common/sqoop/install_sqoop.sh:#!/bin/sh bigtop-packages/src/common/whirr/do-component-build:#!/bin/sh bigtop-packages/src/common/whirr/install_whirr.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/do-component-build:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:echo '#!/bin/bash' > $BUILD_DIR/bin/zkServer-initialize.sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/common/zookeeper/install_zookeeper.sh:#!/bin/sh bigtop-packages/src/deb/flume/flume-node.postinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.postinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.preinst:#!/bin/sh bigtop-packages/src/deb/flume/flume.prerm:#!/bin/sh bigtop-packages/src/deb/giraph/giraph.postinst:#!/bin/sh bigtop-packages/src/deb/giraph/giraph.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/append_licenses.sh:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-conf-pseudo.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-hdfs.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-hdfs.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-httpfs.prerm:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-mapreduce.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-mapreduce.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-yarn.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop-yarn.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.postinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.preinst:#!/bin/sh bigtop-packages/src/deb/hadoop/hadoop.prerm:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.postinst:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.preinst:#!/bin/sh bigtop-packages/src/deb/hbase/hbase.prerm:#!/bin/sh bigtop-packages/src/deb/hbase/install_init_scripts.sh:#!/bin/sh bigtop-packages/src/deb/hbase/service-postinst.tpl:#!/bin/sh bigtop-packages/src/deb/hbase/service-postrm.tpl:#!/bin/sh bigtop-packages/src/deb/hive/hive.postinst:#!/bin/sh bigtop-packages/src/deb/hive/hive.preinst:#!/bin/sh bigtop-packages/src/deb/hive/hive.prerm:#!/bin/sh bigtop-packages/src/deb/hive/install_init_scripts.sh:#!/bin/sh bigtop-packages/src/deb/mahout/mahout.postinst:#!/bin/sh bigtop-packages/src/deb/mahout/mahout.prerm:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.postinst:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.postrm:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.preinst:#!/bin/sh bigtop-packages/src/deb/oozie/oozie.prerm:#!/bin/sh bigtop-packages/src/deb/pig/pig.postinst:#!/bin/sh bigtop-packages/src/deb/pig/pig.preinst:#!/bin/sh bigtop-packages/src/deb/pig/pig.prerm:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop-metastore.postinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop-metastore.preinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop.postinst:#!/bin/sh bigtop-packages/src/deb/sqoop/sqoop.prerm:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.postinst:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.preinst:#!/bin/sh bigtop-packages/src/deb/zookeeper/zookeeper.prerm:#!/bin/sh bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/mku.sh:#!/bin/sh Should we do a blanket search-n-replace? And follow up with updating package dependencies as well?
          Hide
          Eli Collins added a comment -

          +1

          Show
          Eli Collins added a comment - +1
          Roman Shaposhnik made changes -
          Field Original Value New Value
          Description 90% of development and testing of our scripts happens with bash, however, if we don't enforce #!/bin/bash and just specify #!/bin/sh instead on some of the systems we may end up running against #!/bin/dash and other bash alternatives. This could lead to subtle problems. 90% of development and testing of our scripts happens with bash, however, if we don't enforce {noformat}#!/bin/bash{noformat} and just specify {noformat}#!/bin/sh{noformat} instead on some of the systems we may end up running against #!/bin/dash and other bash alternatives. This could lead to subtle problems.
          Roman Shaposhnik created issue -

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development