[SPARK-1257] Endless running task when using pyspark with input file containing a long line

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.1
• Component/s: PySpark
• Assignee: Josh Rosen
• Reporter: Hanchen Su
• Created: 15/Mar/14 04:36
• Updated: 25/Jul/14 21:53
• Resolved: 25/Jul/14 21:53

Description

When launching any pyspark applications with an input file containing a very long line(about 70000 characters), the job will be hanging and never stops. The application UI shows that there is a task running endlessly.

There will be no problem using the scala version with the same input.