[SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.1, 1.1.0
• Assignee: Prashant Sharma
• Reporter: Prashant Sharma
• Created: 11/Jun/14 10:16
• Updated: 10/Dec/15 15:05
• Resolved: 03/Jul/14 22:09

Description

prashant@sc:~/work/spark$ SPARK_MEM=10G bin/pyspark
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/home/prashant/work/spark/python/pyspark/shell.py", line 43, in <module>
sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
File "/home/prashant/work/spark/python/pyspark/context.py", line 94, in _init_
SparkContext._ensure_initialized(self, gateway=gateway)
File "/home/prashant/work/spark/python/pyspark/context.py", line 190, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway()
File "/home/prashant/work/spark/python/pyspark/java_gateway.py", line 51, in launch_gateway
gateway_port = int(proc.stdout.readline())
ValueError: invalid literal for int() with base 10: 'Warning: SPARK_MEM is deprecated, please use a more specific config option\n'

Issue Links

• links to: [Github] Pull Request #1050 (ScrapCodes)