[SPARK-3415] Using sys.stderr in pyspark results in error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.2, 1.1.0
• Fix Version/s: 1.2.0
• Component/s: PySpark
• Labels: python
• Assignee: Unassigned
• Reporter: Ward Viaene
• Created: 05/Sep/14 13:10
• Updated: 08/Sep/14 01:54
• Resolved: 08/Sep/14 01:54

Description

Using sys.stderr in pyspark results in:
File "/home/spark-1.1/dist/python/pyspark/cloudpickle.py", line 660, in save_file
from ..transport.adapter import SerializingAdapter
ValueError: Attempted relative import beyond toplevel package

Code to reproduce (copy paste the code in pyspark):

import sys

class TestClass(object):
def _init_(self, out = sys.stderr):
self.out = out
def getOne(self):
return 'one'

def f():
print type(t)
return 'ok'

t = TestClass()
a = [ 1 , 2, 3, 4, 5 ]
b = sc.parallelize(a)
b.map(lambda x: f()).first()

Issue Links

• links to: Proposed Patch