[SPARK-24740] PySpark tests do not pass with NumPy 0.14.x+

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: ML, PySpark
• Target Version/s: 2.4.0
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 04/Jul/18 15:30
• Updated: 12/Dec/22 18:10
• Resolved: 07/Jul/18 03:39

Description

For example, running PySpark tests failed as below:

**********************************************************************
File "/.../spark/python/pyspark/ml/clustering.py", line 592, in __main__.BisectingKMeans
Failed example:
    model.clusterCenters()[0] == model2.clusterCenters()[0]
Expected:
    array([ True,  True], dtype=bool)
Got:
    array([ True,  True])
**********************************************************************
File "/.../spark/python/pyspark/ml/clustering.py", line 594, in __main__.BisectingKMeans
Failed example:
    model.clusterCenters()[1] == model2.clusterCenters()[1]
Expected:
    array([ True,  True], dtype=bool)
Got:
    array([ True,  True])
**********************************************************************
File "/.../spark/python/pyspark/ml/clustering.py", line 392, in __main__.KMeans
Failed example:
    model.clusterCenters()[0] == model2.clusterCenters()[0]
Expected:
    array([ True,  True], dtype=bool)
Got:
    array([ True,  True])
**********************************************************************
File "/.../spark/python/pyspark/ml/clustering.py", line 394, in __main__.KMeans
Failed example:
    model.clusterCenters()[1] == model2.clusterCenters()[1]
Expected:
    array([ True,  True], dtype=bool)
Got:
    array([ True,  True])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 895, in __main__.DenseMatrix.__str__
Failed example:
    print(dm)
Expected:
    DenseMatrix([[ 0.,  2.],
                 [ 1.,  3.]])
Got:
    DenseMatrix([[0., 2.],
                 [1., 3.]])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 899, in __main__.DenseMatrix.__str__
Failed example:
    print(dm)
Expected:
    DenseMatrix([[ 0.,  1.],
                 [ 2.,  3.]])
Got:
    DenseMatrix([[0., 1.],
                 [2., 3.]])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 939, in __main__.DenseMatrix.toArray
Failed example:
    m.toArray()
Expected:
    array([[ 0.,  2.],
           [ 1.,  3.]])
Got:
    array([[0., 2.],
           [1., 3.]])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 324, in __main__.DenseVector.dot
Failed example:
    dense.dot(np.reshape([1., 2., 3., 4.], (2, 2), order='F'))
Expected:
    array([  5.,  11.])
Got:
    array([ 5., 11.])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 567, in __main__.SparseVector.dot
Failed example:
    a.dot(np.array([[1, 1], [2, 2], [3, 3], [4, 4]]))
Expected:
    array([ 22.,  22.])
Got:
    array([22., 22.])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 895, in __main__.DenseMatrix.__str__
Failed example:
    print(dm)
Expected:
    DenseMatrix([[ 0.,  2.],
                 [ 1.,  3.]])
Got:
    DenseMatrix([[0., 2.],
                 [1., 3.]])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 899, in __main__.DenseMatrix.__str__
Failed example:
    print(dm)
Expected:
    DenseMatrix([[ 0.,  1.],
                 [ 2.,  3.]])
Got:
    DenseMatrix([[0., 1.],
                 [2., 3.]])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 939, in __main__.DenseMatrix.toArray
Failed example:
    m.toArray()
Expected:
    array([[ 0.,  2.],
           [ 1.,  3.]])
Got:
    array([[0., 2.],
           [1., 3.]])
**********************************************************************
File "/.../spark/python/pyspark/ml/linalg/__init__.py", line 567, in __main__.SparseVector.dot
Failed example:
    a.dot(np.array([[1, 1], [2, 2], [3, 3], [4, 4]]))
Expected:
    array([ 22.,  22.])
Got:
    array([22., 22.])

Issue Links

• links to: [Github] Pull Request #21715 (HyukjinKwon)
• links to: GitHub Pull Request #23591