Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2109

Setting SPARK_MEM for bin/pyspark does not work.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.1, 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      prashant@sc:~/work/spark$ SPARK_MEM=10G bin/pyspark
      Python 2.7.6 (default, Mar 22 2014, 22:59:56)
      [GCC 4.8.2] on linux2
      Type "help", "copyright", "credits" or "license" for more information.
      Traceback (most recent call last):
      File "/home/prashant/work/spark/python/pyspark/shell.py", line 43, in <module>
      sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
      File "/home/prashant/work/spark/python/pyspark/context.py", line 94, in _init_
      SparkContext._ensure_initialized(self, gateway=gateway)
      File "/home/prashant/work/spark/python/pyspark/context.py", line 190, in _ensure_initialized
      SparkContext._gateway = gateway or launch_gateway()
      File "/home/prashant/work/spark/python/pyspark/java_gateway.py", line 51, in launch_gateway
      gateway_port = int(proc.stdout.readline())
      ValueError: invalid literal for int() with base 10: 'Warning: SPARK_MEM is deprecated, please use a more specific config option\n'

        Attachments

          Activity

            People

            • Assignee:
              prashant Prashant Sharma
              Reporter:
              prashant Prashant Sharma
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: