[SPARK-18876] An error occurred while trying to connect to the Java server

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.1, 2.0.2
• Component/s: PySpark

Environment

Python 2.7.12

• Assignee: Unassigned
• Reporter: pranludi
• Created: 15/Dec/16 06:42
• Updated: 31/Aug/17 07:08
• Resolved: 15/Dec/16 09:17

Description

I am trying to create spark context object with the following commands on pyspark:

ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35918)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 963, in start
self.socket.connect((self.address, self.port))
File "/usr/local/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/gamedev/spark-2.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 419, in coalesce
File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 1131, in _call_
answer = self.gateway_client.send_command(command)
File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 881, in send_command
connection = self._get_connection()
File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 829, in _get_connection
connection = self._create_connection()
File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 835, in _create_connection
connection.start()
File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 970, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:35918)

---------------------------------------------
I try spark version 2.0.0, 2.0.1, 2.0.2
no problem 2.0.0
but 2.0.1, 2.0.2 occur

python code
------
.....
df = spark.read.json('hdfs://big_big_4000000.json')

json_log = []
for log in df.collect():
jj = {}
try:
for f in log._fields_:
if f == 'I_LogDes':
if log[f] is not None:
log_des_json = json.loads(log[f])
for jf in log_des_json:
json_key = add_2(jf)
if json_key in jj:
json_key = '%s_2' % json_key
jj[json_key] = typeIntStr(log_des_json[jf])
else:
jj[remove_i(f)] = typeIntStr(log[f])
json_log.append(jj)
except:
print log

1. !!! here error occur
   df = spark.read.json(spark.sparkContext.parallelize(json_log))

Issue Links

• duplicates: SPARK-18523 OOM killer may leave SparkContext in broken state causing Connection Refused errors
• duplicates: SPARK-21881 Again: OOM killer may leave SparkContext in broken state causing Connection Refused errors