Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1792

Missing Spark-Shell Configure Options

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • Documentation, Spark Core
    • None

    Description

      The `conf/spark-env.sh.template` does not have configure options for the spark shell. For example to enable Kryo for GraphX when using the spark shell in stand alone mode it appears you must add:

      SPARK_SUBMIT_OPTS="-Dspark.serializer=org.apache.spark.serializer.KryoSerializer "
SPARK_SUBMIT_OPTS+="-Dspark.kryo.registrator=org.apache.spark.graphx.GraphKryoRegistrator  "

      However SPARK_SUBMIT_OPTS is not documented anywhere. Perhaps the spark-shell should have its own options (e.g., SPARK_SHELL_OPTS).

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-30557 Add public documentation for SPARK_SUBMIT_OPTS

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              jegonzal Joseph E. Gonzalez
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: