Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9793

PySpark DenseVector, SparseVector should override __eq__ and __hash__

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.5.0
    • 1.6.0
    • ML, PySpark
    • None

    Description

      See SPARK-9750.
      PySpark DenseVector and SparseVector do not override the equality operator properly. They should use semantics, not representation, for comparison. (This is what Scala currently does.)

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue SPARK-9954 Vector.hashCode should use nonzeros

          • Major - Major loss of function.
          • Resolved
          Is contained by

          Umbrella - An overarching type made of sub-tasks SPARK-9953 ML Vector, Matrix semantic equality + hashcode

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-9098 Inconsistent Dense Vectors hashing between PySpark and Scala

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-9098 Inconsistent Dense Vectors hashing between PySpark and Scala

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-9750 SparseMatrix should override equals

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Pull request #8166 [Github] Pull Request #8166 (yanboliang)

          Activity

            People

              yanboliang Yanbo Liang
              josephkb Joseph K. Bradley
              Xiangrui Meng Xiangrui Meng
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: