Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31698

NPE on big dataset plans

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None
    • Environment:

      AWS EMR: 30 machines, 7TB RAM total.

      Description

      We have big dataset containing 275 SQL operations more than 275 joins.

      On the terminal operation to write data, it fails with NullPointerException.

       

      I understand that such big number of operations might not be what spark is designed for, but NullPointerException is not an ideal way to fail in this case.

       

      For more details, please see the stacktrace.

        Attachments

        1. Spark_NPE_big_dataset.log
          57 kB
          Viacheslav Tradunsky

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Tradunsky Viacheslav Tradunsky
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: