Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9076 Improve NaN value handling
  3. SPARK-8797

Sorting float/double column containing NaNs can lead to "Comparison method violates its general contract!" errors

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.3.0, 1.4.0, 1.5.0
    • 1.5.0
    • SQL
    • None

    Description

      When sorting a float or double column that contains NaN (not a number) values, TimSort may throw a ""Comparison method violates its general contract!" error.

      Attachments

        Issue Links

          Activity

            People

              joshrosen Josh Rosen
              joshrosen Josh Rosen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: