[SPARK-26559] ML image can't work with numpy versions prior to 1.9

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: ML, PySpark
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 07/Jan/19 09:34
• Updated: 12/Dec/22 18:10
• Resolved: 07/Jan/19 10:39

Description

ML image can't work with numpy version prior to 1.9 now.

Current pyspark test can show it:

test_read_images (pyspark.ml.tests.test_image.ImageReaderTest) ... ERROR                                                                              
test_read_images_multiple_times (pyspark.ml.tests.test_image.ImageReaderTest2) ... ok                                                                 
                                                                                                                                                      
======================================================================                                                                                
ERROR: test_read_images (pyspark.ml.tests.test_image.ImageReaderTest)                                                                                 
----------------------------------------------------------------------                                                                                
Traceback (most recent call last):
  File "/Users/viirya/docker_tmp/repos/spark-1/python/pyspark/ml/tests/test_image.py", line 36, in test_read_images                                   
    self.assertEqual(ImageSchema.toImage(array, origin=first_row[0]), first_row)                                                                      
  File "/Users/viirya/docker_tmp/repos/spark-1/python/pyspark/ml/image.py", line 193, in toImage                                                      
    data = bytearray(array.astype(dtype=np.uint8).ravel().tobytes())                                                                                  
AttributeError: 'numpy.ndarray' object has no attribute 'tobytes'                                                                                     
                                                                                                                                                      
----------------------------------------------------------------------
Ran 2 tests in 29.040s                                                                                                                                
                                                                                                                                                      
FAILED (errors=1)

Issue Links

• links to: GitHub Pull Request #23484