[SPARK-49882] Handle or document `NumPy 2.1` difference in Python 3.13

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 4.0.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 04/Oct/24 22:06
• Updated: 23/Oct/24 15:53
• Resolved: 20/Oct/24 05:26

Description

Although SPARK-48710 fixed to use NumPy 2.0 compatible types. Python 3.13 requires NumPy 2.1 (SPARK-49869) and seems to reveal another instances of differences.

New `NumPy 2.1` seems to have a different output style in Python 3.13. Of course, the values are correct.

• https://github.com/apache/spark/actions/runs/11186188886/job/31100777649

  **********************************************************************
  File "/__w/spark/spark/python/pyspark/core/rdd.py", line 2463, in __main__.RDD.sampleStdev
  Failed example:
      sc.parallelize([1, 2, 3]).sampleStdev()
  Expected:
      1.0
  Got:
      np.float64(1.0)
  **********************************************************************
  File "/__w/spark/spark/python/pyspark/core/rdd.py", line 2436, in __main__.RDD.stdev
  Failed example:
      sc.parallelize([1, 2, 3]).stdev()
  Expected:
      0.816...
  Got:
      np.float64(0.816496580927726)
  **********************************************************************
  

Issue Links

• relates to: SPARK-49869 Add NumPy in Python 3.13 image
• relates to: SPARK-48710 Incompatibilities with NumPy 2.0