Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31698

NPE on big dataset plans

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.4.4
    • None
    • Spark Core
    • None
    • AWS EMR: 30 machines, 7TB RAM total.

    Description

      We have big dataset containing 275 SQL operations more than 275 joins.

      On the terminal operation to write data, it fails with NullPointerException.

       

      I understand that such big number of operations might not be what spark is designed for, but NullPointerException is not an ideal way to fail in this case.

       

      For more details, please see the stacktrace.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            Tradunsky Viacheslav Tradunsky
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment