[SPARK-2313] PySpark should accept port via a command line argument rather than STDIN

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: PySpark
• Target Version/s: 1.1.0
• Assignee: Josh Rosen
• Reporter: Patrick Wendell
• Created: 28/Jun/14 01:14
• Updated: 16/Feb/15 23:38
• Resolved: 16/Feb/15 23:38

Description

Relying on stdin is a brittle mechanism and has broken several times in the past. From what I can tell this is used only to bootstrap worker.py one time. It would be strictly simpler to just pass it is a command line.

Issue Links

• is depended upon by: SPARK-1670 PySpark Fails to Create SparkContext Due To Debugging Options in conf/java-opts
• is related to: SPARK-3140 PySpark start-up throws confusing exception
• relates to: SPARK-5810 Maven Coordinate Inclusion failing in pySpark
• relates to: SPARK-2111 pyspark errors when SPARK_PRINT_LAUNCH_COMMAND=1
• links to: [Github] Pull Request #3424 (lvsoft)
• links to: [Github] Pull Request #4603 (JoshRosen)