Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1670

PySpark Fails to Create SparkContext Due To Debugging Options in conf/java-opts

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.3.0
    • PySpark
    • None
    • pats-air:spark pat$ IPYTHON=1 bin/pyspark
      Python 2.7.5 (default, Aug 25 2013, 00:04:04)
      ...
      IPython 1.1.0
      ...
      Spark version 1.0.0-SNAPSHOT

      Using Python version 2.7.5 (default, Aug 25 2013 00:04:04)

    Description

      When JVM debugging options are in conf/java-opts, it causes pyspark to fail when creating the SparkContext. The java-opts file looks like the following:

      -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
      

      Here's the error:

      ---------------------------------------------------------------------------
      ValueError                                Traceback (most recent call last)
      /Library/Python/2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
          202             else:
          203                 filename = fname
      --> 204             __builtin__.execfile(filename, *where)
      
      /Users/pat/Projects/spark/python/pyspark/shell.py in <module>()
           41     SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"])
           42 
      ---> 43 sc = SparkContext(os.environ.get("MASTER", "local[*]"), "PySparkShell", pyFiles=add_files)
           44 
           45 print("""Welcome to
      
      /Users/pat/Projects/spark/python/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway)
           92             tempNamedTuple = namedtuple("Callsite", "function file linenum")
           93             self._callsite = tempNamedTuple(function=None, file=None, linenum=None)
      ---> 94         SparkContext._ensure_initialized(self, gateway=gateway)
           95 
           96         self.environment = environment or {}
      
      /Users/pat/Projects/spark/python/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway)
          172         with SparkContext._lock:
          173             if not SparkContext._gateway:
      --> 174                 SparkContext._gateway = gateway or launch_gateway()
          175                 SparkContext._jvm = SparkContext._gateway.jvm
          176                 SparkContext._writeToFile = SparkContext._jvm.PythonRDD.writeToFile
      
      /Users/pat/Projects/spark/python/pyspark/java_gateway.pyc in launch_gateway()
           44         proc = Popen(command, stdout=PIPE, stdin=PIPE)
           45     # Determine which ephemeral port the server started on:
      ---> 46     port = int(proc.stdout.readline())
           47     # Create a thread to echo output from the GatewayServer, which is required
           48     # for Java log output to show up:
      
      ValueError: invalid literal for int() with base 10: 'Listening for transport dt_socket at address: 5005\n'
      

      Note that when you use JVM debugging, the very first line of output (e.g. when running spark-shell) looks like this:

      Listening for transport dt_socket at address: 5005

      Attachments

        Issue Links

          Activity

            cheffpj Pat McDonough added a comment -

            FYI ahirreddy matei, here's the pyspark issue I was talking to you guys about

            cheffpj Pat McDonough added a comment - FYI ahirreddy matei , here's the pyspark issue I was talking to you guys about

            SPARK-2313 is the root cause of this. a workaround for this would be complex because the extra text on stdout is coming from the same jvm that should produce the py4j port.

            farrellee Matthew Farrellee added a comment - SPARK-2313 is the root cause of this. a workaround for this would be complex because the extra text on stdout is coming from the same jvm that should produce the py4j port.
            joshrosen Josh Rosen added a comment -

            The root cause, SPARK-2313, was fixed for 1.3, so I'm going to mark this as Fixed.

            joshrosen Josh Rosen added a comment - The root cause, SPARK-2313 , was fixed for 1.3, so I'm going to mark this as Fixed.

            People

              Unassigned Unassigned
              cheffpj Pat McDonough
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: