Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24740

PySpark tests do not pass with NumPy 0.14.x+

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: ML, PySpark
    • Labels:
      None
    • Target Version/s:

      Description

      For example, running PySpark tests failed as below:

      **********************************************************************
      File "/.../spark/python/pyspark/ml/clustering.py", line 592, in __main__.BisectingKMeans
      Failed example:
          model.clusterCenters()[0] == model2.clusterCenters()[0]
      Expected:
          array([ True,  True], dtype=bool)
      Got:
          array([ True,  True])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/clustering.py", line 594, in __main__.BisectingKMeans
      Failed example:
          model.clusterCenters()[1] == model2.clusterCenters()[1]
      Expected:
          array([ True,  True], dtype=bool)
      Got:
          array([ True,  True])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/clustering.py", line 392, in __main__.KMeans
      Failed example:
          model.clusterCenters()[0] == model2.clusterCenters()[0]
      Expected:
          array([ True,  True], dtype=bool)
      Got:
          array([ True,  True])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/clustering.py", line 394, in __main__.KMeans
      Failed example:
          model.clusterCenters()[1] == model2.clusterCenters()[1]
      Expected:
          array([ True,  True], dtype=bool)
      Got:
          array([ True,  True])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 895, in __main__.DenseMatrix.__str__
      Failed example:
          print(dm)
      Expected:
          DenseMatrix([[ 0.,  2.],
                       [ 1.,  3.]])
      Got:
          DenseMatrix([[0., 2.],
                       [1., 3.]])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 899, in __main__.DenseMatrix.__str__
      Failed example:
          print(dm)
      Expected:
          DenseMatrix([[ 0.,  1.],
                       [ 2.,  3.]])
      Got:
          DenseMatrix([[0., 1.],
                       [2., 3.]])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 939, in __main__.DenseMatrix.toArray
      Failed example:
          m.toArray()
      Expected:
          array([[ 0.,  2.],
                 [ 1.,  3.]])
      Got:
          array([[0., 2.],
                 [1., 3.]])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 324, in __main__.DenseVector.dot
      Failed example:
          dense.dot(np.reshape([1., 2., 3., 4.], (2, 2), order='F'))
      Expected:
          array([  5.,  11.])
      Got:
          array([ 5., 11.])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 567, in __main__.SparseVector.dot
      Failed example:
          a.dot(np.array([[1, 1], [2, 2], [3, 3], [4, 4]]))
      Expected:
          array([ 22.,  22.])
      Got:
          array([22., 22.])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 895, in __main__.DenseMatrix.__str__
      Failed example:
          print(dm)
      Expected:
          DenseMatrix([[ 0.,  2.],
                       [ 1.,  3.]])
      Got:
          DenseMatrix([[0., 2.],
                       [1., 3.]])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 899, in __main__.DenseMatrix.__str__
      Failed example:
          print(dm)
      Expected:
          DenseMatrix([[ 0.,  1.],
                       [ 2.,  3.]])
      Got:
          DenseMatrix([[0., 1.],
                       [2., 3.]])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 939, in __main__.DenseMatrix.toArray
      Failed example:
          m.toArray()
      Expected:
          array([[ 0.,  2.],
                 [ 1.,  3.]])
      Got:
          array([[0., 2.],
                 [1., 3.]])
      **********************************************************************
      File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 567, in __main__.SparseVector.dot
      Failed example:
          a.dot(np.array([[1, 1], [2, 2], [3, 3], [4, 4]]))
      Expected:
          array([ 22.,  22.])
      Got:
          array([22., 22.])
      

        Attachments

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              hyukjin.kwon Hyukjin Kwon
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: