Details

    • Sub-task
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.6.0
    • 1.8.0
    • Server
    • None

    Description

      Noticed the following in drillbit.sh:

      1) Comment: DRILL_LOG_DIR Where log files are stored. PWD by default.
      Code: DRILL_LOG_DIR=/var/log/drill or, if it does not exist, $DRILL_HOME/log

      2) Comment: DRILL_PID_DIR The pid files are stored. /tmp by default.
      Code: DRILL_PID_DIR=$DRILL_HOME

      3) Redundant checking of JAVA_HOME. drillbit.sh sources drill-config.sh which checks JAVA_HOME. Later, drillbit.sh checks it again. The second check is both unnecessary and prints a less informative message than the drill-config.sh check. Suggestion: Remove the JAVA_HOME check in drillbit.sh.

      4) Though drill-config.sh carefully checks JAVA_HOME, it does not export the JAVA_HOME variable. Perhaps this is why drillbit.sh repeats the check? Recommended: export JAVA_HOME from drill-config.sh.

      5) Both drillbit.sh and the sourced drill-config.sh check DRILL_LOG_DIR and set the default value. Drill-config.sh defaults to /var/log/drill, or if that fails, to $DRILL_HOME/log. Drillbit.sh just sets /var/log/drill and does not handle the case where that directory is not writable. Suggested: remove the check in drillbit.sh.

      6) Drill-config.sh checks the writability of the DRILL_LOG_DIR by touching sqlline.log, but does not delete that file, leaving a bogus, empty client log file on the drillbit server. Recommendation: use bash commands instead.

      7) The implementation of the above check is a bit awkward. It has a fallback case with somewhat awkward logic. Clean this up.

      8) drillbit.sh, but not drill-config.sh, attempts to create /var/log/drill if it does not exist. Recommended: decide on a single choice, implement it in drill-config.sh.

      9) drill-config.sh checks if $DRILL_CONF_DIR is a directory. If not, defaults it to $DRILL_HOME/conf. This can lead to subtle errors. If I use
      drillbit.sh --config /misspelled/path
      where I mistype the path, I won't get an error, I get the default config, which may not at all be what I want to run. Recommendation: if the value of DRILL_CONF_DRILL is passed into the script (as a variable or via --config), then that directory must exist. Else, use the default.

      10) drill-config.sh exports, but may not set, HADOOP_HOME. This may be left over from the original Hadoop script that the Drill script was based upon. Recomendation: export only in the case that HADOOP_HOME is set for cygwin.

      11) Drill-config.sh checks JAVA_HOME and prints a big, bold error message to stderr if JAVA_HOME is not set. Then, it checks the Java version and prints a different message (to stdout) if the version is wrong. Recommendation: use the same format (and stderr) for both.

      12) Similarly, other Java checks later in the script produce messages to stdout, not stderr.

      13) Drill-config.sh searches $JAVA_HOME to find java/java.exe and verifies that it is executable. The script then throws away what we just found. Then, drill-bit.sh tries to recreate this information as:
      JAVA=$JAVA_HOME/bin/java
      This is wrong in two ways: 1) it ignores the actual java location and assumes it, and 2) it does not handle the java.exe case that drill-config.sh carefully worked out.
      Recommendation: export JAVA from drill-config.sh and remove the above line from drillbit.sh.

      14) drillbit.sh presumably takes extra arguments like this:
      drillbit.sh -Dvar0=value0 --config /my/conf/dir start -Dvar1=value1 -Dvar2=value2 -Dvar3=value3
      The -D bit allows the user to override config variables at the command line. But, the scripts don't use the values.
      A) drill-config.sh consumes --config /my/conf/dir after consuming the leading arguments:
      while [ $# -gt 1 ]; do
      if [ "--config" = "$1" ]; then
      shift
      confdir=$1
      shift
      DRILL_CONF_DIR=$confdir
      else

      1. Presume we are at end of options and break
        break
        fi
        done
        B) drill-bit.sh will discard the var1:
        startStopStatus=$1 <-- grabs "start"
        shift
        command=drillbit
        shift <-- Consumes -Dvar1=value1
        C) Remaining values passed back into drillbit.sh:
        args=$@
        nohup $thiscmd internal_start $command $args
        D) Second invocation discards -Dvar2=value2 as described above.
        E) Remaining values are passed to runbit:
        "$DRILL_HOME"/bin/runbit $command "$@" start
        F) Where they again pass though drill-config. (Allowing us to do:
        drillbit.sh --config /first/conf --config /second/conf
        which is asking for trouble)
        G) And, the remaining arguments are simply not used:
        exec $JAVA -Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=$DRILLBIT_QUERY_LOG_PATH $DRILL_ALL_JAVA_OPTS -cp $CP org.apache.drill.exec.server.Drillbit

      15) The checking of command-line args in drillbit.sh is wrong:

      1. if no args specified, show usage
        if [ $# -lt 1 ]; then
        echo $usage
        exit 1
        fi
        ...
        . "$bin"/drill-config.sh

      But, note, that drill-config.sh handles:
      drillbit.sh --config /conf/dir
      Consuming those two arguments will leave no command argument. Thus, the no-argument check should be done AFTER consuming --config.

      16) As noted above, both drillbit.sh and runbit source drill-config.sh. But runbit is (apparently) only every called from drillbit.sh. Therefore, we do the same setup & check tasks twice. (In addition to reading --config twice as noted above.) Recommendation: omit the sourcing of drill-config.sh in runbit.

      17) The name of the drillbit.sh script is used in many messages. It is hardcoded, but appears to originally have been taken from $0:

      command=drillbit
      shift

      Recommended: get the name from the script, strip the directory, and strip the suffix. So, /foo/drillbit2.sh becomes drillbit2, \windir\drillbit2.bat becomes drillbit2.

      18) drillbit.sh creates a pid file to record the pid of the running Drillbit. However, the file is not deleted upon normal Drill exit. Better would be to remove this file on exit to keep things tidy.

      19) Similarly, when the stop command detects a pid file, but no running process with that pid, it prints a message saying so, but does not clean up the (unwanted) pid file. It should do so.

      20) The runbit script sets up Java options as follows:

      DRILL_ALL_JAVA_OPTS="$DRILLBIT_JAVA_OPTS $DRILL_JAVA_OPTS $SERVER_GC_OPTS"

      Presumably the DRILL_JAVA_OPTS are for all Drill apps (including the client) while DRILLBIT_JAVA_OPTS are for the server (drillbit).
      Since later Java options override earlier ones, more specific options (for the server) should come after more general ones (for all of Drill). So, order should be:

      DRILL_ALL_JAVA_OPTS="$DRILL_JAVA_OPTS $DRILLBIT_JAVA_OPTS $SERVER_GC_OPTS"

      21) The startup scripts go to great lengths to properly set up the logs. But, we allow drill-env.sh to override them.

      exec $JAVA -Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=$DRILLBIT_QUERY_LOG_PATH $DRILL_ALL_JAVA_OPTS ...

      The computed log properties come before the user-defined properties, and so user defined properties can override those log settings. The computed log settings are used to write log entries and should be considered "correct."
      Recommended: change option order as follows:

      exec $JAVA $DRILL_ALL_JAVA_OPTS -Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=$DRILLBIT_QUERY_LOG_PATH ...

      22) As above, but in sqlline:

      exec "$JAVA" $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS

      Should be:

      exec "$JAVA" $DRILL_JAVA_OPTS $DRILL_SHELL_JAVA_OPTS

      23) The implementations of each command unnecessarily pass along $command on internal calls:

      (start)
      nohup $thiscmd internal_start $command $args < /dev/null >> ${logout} 2>&1 &
      (internal_start)
      nice -n $DRILL_NICENESS "$DRILL_HOME"/bin/runbit \
      $command "$@" start >> "$logout" 2>&1 &

      Results in "drillbit drillbit start" being passed to runbit. Today, runbit does not use these arguments. But, if it did, it would get unncesssary clutter.

      24) Benign. The restart command passes $command uncessarily when calling the drillbit.sh script recursively:
      $thiscmd --config "${DRILL_CONF_DIR}" stop $command $args &
      $thiscmd --config "${DRILL_CONF_DIR}" start $command $args &

      results in drillbit.sh --conf ... stop drill bit

      25) The stop command removes a file using an undefined variable:
      rm -f "$DRILL_START_FILE"

      This variable is never defined in any of the startup scripts. If the user can define it, we should be checking if the variable is empty. Suggestion: remove this line.

      Attachments

        Activity

          paul-rogers Paul Rogers added a comment -

          The script has some iffy behavior. If the requested log directory (set with DRILL_LOG_DIR) does not exist or is not writable, the log directory defaults to $DRILL_HOME/log.

          This should happen only if DRILL_LOG_DIR is not set and /var/log/drill does not exist. Otherwise, the user said we should use the given directory; give an error if the directory does not exist Drill should try to create it. If that fails (or the directory is not writable), then fail with an error.

          paul-rogers Paul Rogers added a comment - The script has some iffy behavior. If the requested log directory (set with DRILL_LOG_DIR) does not exist or is not writable, the log directory defaults to $DRILL_HOME/log. This should happen only if DRILL_LOG_DIR is not set and /var/log/drill does not exist. Otherwise, the user said we should use the given directory; give an error if the directory does not exist Drill should try to create it. If that fails (or the directory is not writable), then fail with an error.
          paul-rogers Paul Rogers added a comment - - edited

          Minor point. When checking status using drillbit.sh, include the pid of the running process:
          drillbit is running, pid: 78390

          Also, add the uptime:
          drillbit is running, pid: 78390, uptime: 24 hours

          paul-rogers Paul Rogers added a comment - - edited Minor point. When checking status using drillbit.sh, include the pid of the running process: drillbit is running, pid: 78390 Also, add the uptime: drillbit is running, pid: 78390, uptime: 24 hours
          paul-rogers Paul Rogers added a comment -

          Not a bug, but the drill-config.sh script sources the $DRILL_CONF_DIR/drill-env.sh script without checking if it exists. Since this file is optional, the script should skip sourcing the file if it does not exist.

          paul-rogers Paul Rogers added a comment - Not a bug, but the drill-config.sh script sources the $DRILL_CONF_DIR/drill-env.sh script without checking if it exists. Since this file is optional, the script should skip sourcing the file if it does not exist.
          githubbot ASF GitHub Bot added a comment -

          GitHub user paul-rogers opened a pull request:

          https://github.com/apache/drill/pull/477

          Drill 4581

          Provides fixes to the Drill launch scripts for multiple issues in DRILL-4581.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/paul-rogers/drill DRILL-4581-Fix

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/drill/pull/477.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #477


          commit abbfe84e35517c37f59a507694a4f0224137d2b8
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-08T21:04:40Z

          Merge remote-tracking branch 'apache/master'

          commit e68ab2d8c50ce4d64631f410cbcbe7f4e98aeb8c
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-13T23:27:06Z

          Merge remote-tracking branch 'apache/master'

          commit 6b4bec80e5473f10c05006da58a32bf9bd3f6b13
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-14T00:06:06Z

          Fixed issues from DRILL-4581


          githubbot ASF GitHub Bot added a comment - GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/477 Drill 4581 Provides fixes to the Drill launch scripts for multiple issues in DRILL-4581 . You can merge this pull request into a Git repository by running: $ git pull https://github.com/paul-rogers/drill DRILL-4581 -Fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/477.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #477 commit abbfe84e35517c37f59a507694a4f0224137d2b8 Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-08T21:04:40Z Merge remote-tracking branch 'apache/master' commit e68ab2d8c50ce4d64631f410cbcbe7f4e98aeb8c Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-13T23:27:06Z Merge remote-tracking branch 'apache/master' commit 6b4bec80e5473f10c05006da58a32bf9bd3f6b13 Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-14T00:06:06Z Fixed issues from DRILL-4581
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers closed the pull request at:

          https://github.com/apache/drill/pull/477

          githubbot ASF GitHub Bot added a comment - Github user paul-rogers closed the pull request at: https://github.com/apache/drill/pull/477
          githubbot ASF GitHub Bot added a comment -

          GitHub user paul-rogers opened a pull request:

          https://github.com/apache/drill/pull/478

          DRILL-4581: Minor fixes to drill startup scripts for linux

          This pull request replaces a previous one. No code differences; this one resolves some git issues.

          Provides fixes to a number of small problems as explained in DRILL-4581. This work is in preparation for Drill-on-YARN, and contains a new drillbit.sh option to launch Drill under YARN. Otherwise, the public interface to the scripts remains unchanged. Features that used to work still work. Features that were broken now work as originally intended.

          Does not contain changes for the proposed “site” config extension.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/paul-rogers/drill DRILL-4581-2

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/drill/pull/478.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #478


          commit abbfe84e35517c37f59a507694a4f0224137d2b8
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-08T21:04:40Z

          Merge remote-tracking branch 'apache/master'

          commit e68ab2d8c50ce4d64631f410cbcbe7f4e98aeb8c
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-13T23:27:06Z

          Merge remote-tracking branch 'apache/master'

          commit 3289e5840d54859b5d364101e1a8ae09d135a717
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-14T17:35:09Z

          DRILL-4581: Multiple fixes to Drill launch scripts

          Fixes to a number of small problems as explained in DRILL-4581. This
          work is in preparation for Drill-on-YARN, and contains a new
          drillbit.sh option to launch Drill under YARN. Otherwise, the public
          interface to the scripts remains unchanged. Features that used to work
          still work. Features that were broken now work as originally intended.
          Does not contain changes for the proposed “site” config extension.


          githubbot ASF GitHub Bot added a comment - GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/478 DRILL-4581 : Minor fixes to drill startup scripts for linux This pull request replaces a previous one. No code differences; this one resolves some git issues. Provides fixes to a number of small problems as explained in DRILL-4581 . This work is in preparation for Drill-on-YARN, and contains a new drillbit.sh option to launch Drill under YARN. Otherwise, the public interface to the scripts remains unchanged. Features that used to work still work. Features that were broken now work as originally intended. Does not contain changes for the proposed “site” config extension. You can merge this pull request into a Git repository by running: $ git pull https://github.com/paul-rogers/drill DRILL-4581 -2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/478.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #478 commit abbfe84e35517c37f59a507694a4f0224137d2b8 Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-08T21:04:40Z Merge remote-tracking branch 'apache/master' commit e68ab2d8c50ce4d64631f410cbcbe7f4e98aeb8c Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-13T23:27:06Z Merge remote-tracking branch 'apache/master' commit 3289e5840d54859b5d364101e1a8ae09d135a717 Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-14T17:35:09Z DRILL-4581 : Multiple fixes to Drill launch scripts Fixes to a number of small problems as explained in DRILL-4581 . This work is in preparation for Drill-on-YARN, and contains a new drillbit.sh option to launch Drill under YARN. Otherwise, the public interface to the scripts remains unchanged. Features that used to work still work. Features that were broken now work as originally intended. Does not contain changes for the proposed “site” config extension.
          paul-rogers Paul Rogers added a comment -

          The above issues will be addressed as part of the Drill-on-YARN project to allow that project to reuse the existing scripts.

          paul-rogers Paul Rogers added a comment - The above issues will be addressed as part of the Drill-on-YARN project to allow that project to reuse the existing scripts.
          paul-rogers Paul Rogers added a comment -

          Another issue found by John O., who was trying to enable GC logging.

          Drill-env.sh has the following:

          export SERVER_GC_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseG1GC "

          Presumably, to enable logging, users add teh following

          export SERVER_GC_OPTS="$SERVER_GC_OPTS -Xloggc:<FILE-PATH>"

          Which is supposed to be replaced with teh actual log file path in drillbit.sh:

          export SERVER_GC_OPTS=${SERVER_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${loggc}"}

          The substitution here works only if SERVER_GC_OPTS contains ONLY the -Xloggc stuff, but fails when it contains the extra material.

          paul-rogers Paul Rogers added a comment - Another issue found by John O., who was trying to enable GC logging. Drill-env.sh has the following: export SERVER_GC_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseG1GC " Presumably, to enable logging, users add teh following export SERVER_GC_OPTS="$SERVER_GC_OPTS -Xloggc:<FILE-PATH>" Which is supposed to be replaced with teh actual log file path in drillbit.sh: export SERVER_GC_OPTS=${SERVER_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${loggc}"} The substitution here works only if SERVER_GC_OPTS contains ONLY the -Xloggc stuff, but fails when it contains the extra material.
          githubbot ASF GitHub Bot added a comment -

          Github user shishkin commented on a diff in the pull request:

          https://github.com/apache/drill/pull/478#discussion_r69077287

          — Diff: distribution/src/resources/runbit —
          @@ -15,11 +15,41 @@

          1. See the License for the specific language governing permissions and
          2. limitations under the License.

          bin=`dirname "${BASH_SOURCE$0}"`
          -bin=`cd "$bin">/dev/null; pwd`
          +# Invoke Drill using Java. Command line arguments are assumed to be
          +# of the form
          +# -Dname=value
          +# and are passed to Java as JVM arguments.
          +#
          +# Environment Variables
          +#
          +# DRILLBIT_LOG_PATH Path to the Drillbit log file.
          +# (Set in drillbit.sh)
          +# DRILLBIT_QUERY_LOG_PATH Path the the query log file.
          +# (Set in drillbit.sh)
          +# DRILL_JAVA_OPTS Optional JVM arguments such as system
          +# property overides used by both the
          +# drillbit (server) and client,
          +# set in drill-env.sh or the environment.
          +# DRILLBIT_JAVA_OPTS Optional JVM arguments specifically for
          +# the server (drillbit). Set in the
          +# environment or in the user defined
          +# drill-env.sh
          +# SERVER_GC_OPTS Set in drill-env.sh, customized by
          +# drillbit.sh.
          +# CP Drillbit classpath set in drillbit.sh
          +
          +#bin=`dirname "${BASH_SOURCE-$0}"`
          +#bin=`cd "$bin">/dev/null; pwd`
          +
          +LOG_OPTS="-Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=$DRILLBIT_QUERY_LOG_PATH"
          +DRILL_ALL_JAVA_OPTS="$DRILL_JAVA_OPTS $DRILLBIT_JAVA_OPTS $SERVER_GC_OPTS $@ $LOG_OPTS"
          — End diff –

          Glad to see `$@` in there. Finally there would be an option to override memory settings from the command line rather than through files. This will simplify Drill deployment on Mesos with Marathon.

          githubbot ASF GitHub Bot added a comment - Github user shishkin commented on a diff in the pull request: https://github.com/apache/drill/pull/478#discussion_r69077287 — Diff: distribution/src/resources/runbit — @@ -15,11 +15,41 @@ See the License for the specific language governing permissions and limitations under the License. bin=`dirname "${BASH_SOURCE $0}"` -bin=`cd "$bin">/dev/null; pwd` +# Invoke Drill using Java. Command line arguments are assumed to be +# of the form +# -Dname=value +# and are passed to Java as JVM arguments. +# +# Environment Variables +# +# DRILLBIT_LOG_PATH Path to the Drillbit log file. +# (Set in drillbit.sh) +# DRILLBIT_QUERY_LOG_PATH Path the the query log file. +# (Set in drillbit.sh) +# DRILL_JAVA_OPTS Optional JVM arguments such as system +# property overides used by both the +# drillbit (server) and client, +# set in drill-env.sh or the environment. +# DRILLBIT_JAVA_OPTS Optional JVM arguments specifically for +# the server (drillbit). Set in the +# environment or in the user defined +# drill-env.sh +# SERVER_GC_OPTS Set in drill-env.sh, customized by +# drillbit.sh. +# CP Drillbit classpath set in drillbit.sh + +#bin=`dirname "${BASH_SOURCE-$0}"` +#bin=`cd "$bin">/dev/null; pwd` + +LOG_OPTS="-Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=$DRILLBIT_QUERY_LOG_PATH" +DRILL_ALL_JAVA_OPTS="$DRILL_JAVA_OPTS $DRILLBIT_JAVA_OPTS $SERVER_GC_OPTS $@ $LOG_OPTS" — End diff – Glad to see `$@` in there. Finally there would be an option to override memory settings from the command line rather than through files. This will simplify Drill deployment on Mesos with Marathon.
          githubbot ASF GitHub Bot added a comment -

          GitHub user paul-rogers opened a pull request:

          https://github.com/apache/drill/pull/544

          DRILL 4581 Revised

          Extensive revisions to the Drill scripts to prepare them for
          Drill-on-YARN, to provide a full site directory, to move
          distribution-specific settings out of the user’s drill-env.sh file, and
          to fix the bugs outlined in DRILL-4581.

          Contains only Drill script fixes; does not contain the new
          Drill-on-YARN scripts.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/paul-rogers/drill DRILL-4581-Revised

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/drill/pull/544.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #544


          commit abbfe84e35517c37f59a507694a4f0224137d2b8
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-08T21:04:40Z

          Merge remote-tracking branch 'apache/master'

          commit e68ab2d8c50ce4d64631f410cbcbe7f4e98aeb8c
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-04-13T23:27:06Z

          Merge remote-tracking branch 'apache/master'

          commit ce7d43be1808690f80f6a3c0a826ffb2725ef817
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-07-11T02:55:09Z

          Merge remote-tracking branch 'apache/master'

          commit 291f28219fa11ab317ac4b0923e68502881c0b07
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-07-12T23:19:28Z

          DRILL-4581 - Drill script fixes

          Extensive revisions to the Drill scripts to prepare them for
          Drill-on-YARN, to provide a full site directory, to move
          distribution-specific settings out of the user’s drill-env.sh file, and
          to fix the bugs outlined in DRILL-4581.

          Contains only Drill script fixes; does not contain the new
          Drill-on-YARN scripts.


          githubbot ASF GitHub Bot added a comment - GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/544 DRILL 4581 Revised Extensive revisions to the Drill scripts to prepare them for Drill-on-YARN, to provide a full site directory, to move distribution-specific settings out of the user’s drill-env.sh file, and to fix the bugs outlined in DRILL-4581 . Contains only Drill script fixes; does not contain the new Drill-on-YARN scripts. You can merge this pull request into a Git repository by running: $ git pull https://github.com/paul-rogers/drill DRILL-4581 -Revised Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/544.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #544 commit abbfe84e35517c37f59a507694a4f0224137d2b8 Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-08T21:04:40Z Merge remote-tracking branch 'apache/master' commit e68ab2d8c50ce4d64631f410cbcbe7f4e98aeb8c Author: Paul Rogers <progers@maprtech.com> Date: 2016-04-13T23:27:06Z Merge remote-tracking branch 'apache/master' commit ce7d43be1808690f80f6a3c0a826ffb2725ef817 Author: Paul Rogers <progers@maprtech.com> Date: 2016-07-11T02:55:09Z Merge remote-tracking branch 'apache/master' commit 291f28219fa11ab317ac4b0923e68502881c0b07 Author: Paul Rogers <progers@maprtech.com> Date: 2016-07-12T23:19:28Z DRILL-4581 - Drill script fixes Extensive revisions to the Drill scripts to prepare them for Drill-on-YARN, to provide a full site directory, to move distribution-specific settings out of the user’s drill-env.sh file, and to fix the bugs outlined in DRILL-4581 . Contains only Drill script fixes; does not contain the new Drill-on-YARN scripts.
          githubbot ASF GitHub Bot added a comment -

          GitHub user paul-rogers opened a pull request:

          https://github.com/apache/drill/pull/547

          Drill-4581: Extensive revisions to the Drill launch scripts.

          See DRILL-4581 and DRILL-4591 for an overview.
          See DRILL-4581 for a detailed list of bugs fixed.
          See DRILL-4591 for the motivation for the new "site" directory support.
          Changes support DRILL-1170 (Drill-on-YARN).

          Broad overview of changes:

          • Extended the existing "config" directory concept to create a
            "site" directory that holds all site-specific files, leaving the
            Drill directory ($DRILL_HOME) to contain only Drill-provided files.
            The site directory is handy for all Drill users because it eases
            upgrades, but is necessary to simplify Drill-on-YARN deployments.
            Use the --config (for backward compatibility) or --site (more
            descriptive) option to point to the site directory.
          • Moved distribution-specific settings, and Drill defaults, out of
            drill-env.sh. Now, drill-env.sh contains only user settings, avoiding
            the need to do multi-way merges on upgrades. Distribution-specific
            files now reside in a new $DRILL_HOME/conf/distrib-env.sh file.
          • Refactored the launch scripts to allow the bulk of setup to be shared
            between the "classic" Drill deamon script (drillbit.sh) and the new
            Drill-on-YARN scripts.
          • Added a new "run" option to drillbit.sh to allow Drill to run as a
            child process as needed by tools such as Mesos.
          • Changes ensure backward compatibility. Users of earlier releases can
            upgrade to the release with this fix without doing anything special.
            Drill will "just work." However, users can optionally clean up the
            drill-env.sh script, optionally use the site directory, and so on.
            Hoever these upgrades are not required.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/paul-rogers/drill DRILL-4581-PR3

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/drill/pull/547.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #547


          commit d9e0e4561d21a3dac3d582254a6b1c9885a6e196
          Author: Paul Rogers <progers@maprtech.com>
          Date: 2016-07-12T23:19:28Z

          Drill-4581: Extensive revisions to the Drill launch scripts.

          See DRILL-4581 and DRILL-4591 for an overview.
          See DRILL-4581 for a detailed list of bugs fixed.
          See DRILL-4591 for the motivation for the new "site" directory support.
          Changes support DRILL-1170 (Drill-on-YARN).

          Broad overview of changes:

          • Extended the existing "config" directory concept to create a
            "site" directory that holds all site-specific files, leaving the
            Drill directory ($DRILL_HOME) to contain only Drill-provided files.
            The site directory is handy for all Drill users because it eases
            upgrades, but is necessary to simplify Drill-on-YARN deployments.
            Use the --config (for backward compatibility) or --site (more
            descriptive) option to point to the site directory.
          • Moved distribution-specific settings, and Drill defaults, out of
            drill-env.sh. Now, drill-env.sh contains only user settings, avoiding
            the need to do multi-way merges on upgrades. Distribution-specific
            files now reside in a new $DRILL_HOME/conf/distrib-env.sh file.
          • Refactored the launch scripts to allow the bulk of setup to be shared
            between the "classic" Drill deamon script (drillbit.sh) and the new
            Drill-on-YARN scripts.
          • Added a new "run" option to drillbit.sh to allow Drill to run as a
            child process as needed by tools such as Mesos.
          • Changes ensure backward compatibility. Users of earlier releases can
            upgrade to the release with this fix without doing anything special.
            Drill will "just work." However, users can optionally clean up the
            drill-env.sh script, optionally use the site directory, and so on.
            Hoever these upgrades are not required.

          githubbot ASF GitHub Bot added a comment - GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/547 Drill-4581: Extensive revisions to the Drill launch scripts. See DRILL-4581 and DRILL-4591 for an overview. See DRILL-4581 for a detailed list of bugs fixed. See DRILL-4591 for the motivation for the new "site" directory support. Changes support DRILL-1170 (Drill-on-YARN). Broad overview of changes: Extended the existing "config" directory concept to create a "site" directory that holds all site-specific files, leaving the Drill directory ($DRILL_HOME) to contain only Drill-provided files. The site directory is handy for all Drill users because it eases upgrades, but is necessary to simplify Drill-on-YARN deployments. Use the --config (for backward compatibility) or --site (more descriptive) option to point to the site directory. Moved distribution-specific settings, and Drill defaults, out of drill-env.sh. Now, drill-env.sh contains only user settings, avoiding the need to do multi-way merges on upgrades. Distribution-specific files now reside in a new $DRILL_HOME/conf/distrib-env.sh file. Refactored the launch scripts to allow the bulk of setup to be shared between the "classic" Drill deamon script (drillbit.sh) and the new Drill-on-YARN scripts. Added a new "run" option to drillbit.sh to allow Drill to run as a child process as needed by tools such as Mesos. Changes ensure backward compatibility. Users of earlier releases can upgrade to the release with this fix without doing anything special. Drill will "just work." However, users can optionally clean up the drill-env.sh script, optionally use the site directory, and so on. Hoever these upgrades are not required. You can merge this pull request into a Git repository by running: $ git pull https://github.com/paul-rogers/drill DRILL-4581 -PR3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/547.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #547 commit d9e0e4561d21a3dac3d582254a6b1c9885a6e196 Author: Paul Rogers <progers@maprtech.com> Date: 2016-07-12T23:19:28Z Drill-4581: Extensive revisions to the Drill launch scripts. See DRILL-4581 and DRILL-4591 for an overview. See DRILL-4581 for a detailed list of bugs fixed. See DRILL-4591 for the motivation for the new "site" directory support. Changes support DRILL-1170 (Drill-on-YARN). Broad overview of changes: Extended the existing "config" directory concept to create a "site" directory that holds all site-specific files, leaving the Drill directory ($DRILL_HOME) to contain only Drill-provided files. The site directory is handy for all Drill users because it eases upgrades, but is necessary to simplify Drill-on-YARN deployments. Use the --config (for backward compatibility) or --site (more descriptive) option to point to the site directory. Moved distribution-specific settings, and Drill defaults, out of drill-env.sh. Now, drill-env.sh contains only user settings, avoiding the need to do multi-way merges on upgrades. Distribution-specific files now reside in a new $DRILL_HOME/conf/distrib-env.sh file. Refactored the launch scripts to allow the bulk of setup to be shared between the "classic" Drill deamon script (drillbit.sh) and the new Drill-on-YARN scripts. Added a new "run" option to drillbit.sh to allow Drill to run as a child process as needed by tools such as Mesos. Changes ensure backward compatibility. Users of earlier releases can upgrade to the release with this fix without doing anything special. Drill will "just work." However, users can optionally clean up the drill-env.sh script, optionally use the site directory, and so on. Hoever these upgrades are not required.
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers closed the pull request at:

          https://github.com/apache/drill/pull/544

          githubbot ASF GitHub Bot added a comment - Github user paul-rogers closed the pull request at: https://github.com/apache/drill/pull/544
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/drill/pull/547

          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/547

          Fixed in 1f3c74d.

          sudheeshkatkam Sudheesh Katkam added a comment - Fixed in 1f3c74d .
          sollala_drill Suresh Ollala added a comment -

          agirishassigning for verification

          sollala_drill Suresh Ollala added a comment - agirish assigning for verification
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers commented on the issue:

          https://github.com/apache/drill/pull/478

          Replaced by later commit.

          githubbot ASF GitHub Bot added a comment - Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/478 Replaced by later commit.
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers closed the pull request at:

          https://github.com/apache/drill/pull/478

          githubbot ASF GitHub Bot added a comment - Github user paul-rogers closed the pull request at: https://github.com/apache/drill/pull/478

          People

            paul-rogers Paul Rogers
            paul-rogers Paul Rogers
            Abhishek Girish Abhishek Girish
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: