Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: spark
    • Labels:
      None

      Description

      Current spark-env.sh is a bit complex; It's not easy to read and edit. It would be nice to have a simplified spark-env.sh

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user youngwookim opened a pull request:

          https://github.com/apache/bigtop/pull/31

          BIGTOP-1974. Revise SPARK_HOME/conf/spark-env.sh

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/youngwookim/bigtop BIGTOP-1974

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/31.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #31


          commit 542a2b08b73ba905b58f0a61949095d4f27d49b0
          Author: YoungWoo Kim <ywkim@apache.org>
          Date: 2015-08-20T10:53:43Z

          BIGTOP-1974. Revise SPARK_HOME/conf/spark-env.sh


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/31 BIGTOP-1974 . Revise SPARK_HOME/conf/spark-env.sh You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1974 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/31.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #31 commit 542a2b08b73ba905b58f0a61949095d4f27d49b0 Author: YoungWoo Kim <ywkim@apache.org> Date: 2015-08-20T10:53:43Z BIGTOP-1974 . Revise SPARK_HOME/conf/spark-env.sh
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          First PR: Removed deprecated or unused env vars from spark-env.sh
          As a result, these vars will remain by default:

          export HADOOP_HOME=${HADOOP_HOME:-/usr/lib/hadoop}
          export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}
          
          export SPARK_MASTER_IP=`hostname`
          export SPARK_MASTER_WEBUI_PORT=18080
          export SPARK_MASTER_PORT=7077
          export SPARK_WORKER_PORT=7078
          export SPARK_WORKER_WEBUI_PORT=18081
          export SPARK_WORKER_DIR=/var/run/spark/work
          export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=hdfs:///var/log/spark/apps -Dspark.history.ui.port=18082"
          
          export SPARK_LOG_DIR=/var/log/spark
          
          export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${HADOOP_HOME}/lib/native
          
          Show
          warwithin YoungWoo Kim added a comment - - edited First PR: Removed deprecated or unused env vars from spark-env.sh As a result, these vars will remain by default: export HADOOP_HOME=${HADOOP_HOME:-/usr/lib/hadoop} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf} export SPARK_MASTER_IP=`hostname` export SPARK_MASTER_WEBUI_PORT=18080 export SPARK_MASTER_PORT=7077 export SPARK_WORKER_PORT=7078 export SPARK_WORKER_WEBUI_PORT=18081 export SPARK_WORKER_DIR=/ var /run/spark/work export SPARK_HISTORY_OPTS= "$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=hdfs: /// var /log/spark/apps -Dspark.history.ui.port=18082" export SPARK_LOG_DIR=/ var /log/spark export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${HADOOP_HOME}/lib/ native
          Hide
          evans_ye Evans Ye added a comment -

          +1. The patch looks good to me. If you've tested it, please commit.

          Show
          evans_ye Evans Ye added a comment - +1. The patch looks good to me. If you've tested it, please commit.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rnowling commented on the pull request:

          https://github.com/apache/bigtop/pull/31#issuecomment-134473321

          If tested, +1 LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user rnowling commented on the pull request: https://github.com/apache/bigtop/pull/31#issuecomment-134473321 If tested, +1 LGTM
          Hide
          warwithin YoungWoo Kim added a comment -

          Thanks Evans Ye and RJ Nowling for reviewing the patch. Will commit this shortly and file remaining JIRAs for Spark

          Show
          warwithin YoungWoo Kim added a comment - Thanks Evans Ye and RJ Nowling for reviewing the patch. Will commit this shortly and file remaining JIRAs for Spark
          Hide
          warwithin YoungWoo Kim added a comment -
          Show
          warwithin YoungWoo Kim added a comment - Filed BIGTOP-1987 and BIGTOP-1988

            People

            • Assignee:
              warwithin YoungWoo Kim
              Reporter:
              warwithin YoungWoo Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development