Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1257

Endless running task when using pyspark with input file containing a long line

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.9.0
    • 0.9.1
    • PySpark
    • None

    Description

      When launching any pyspark applications with an input file containing a very long line(about 70000 characters), the job will be hanging and never stops. The application UI shows that there is a task running endlessly.

      There will be no problem using the scala version with the same input.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            joshrosen Josh Rosen
            Hanchen Hanchen Su
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment