Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26559

ML image can't work with numpy versions prior to 1.9

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.4.1, 3.0.0
    • Component/s: ML, PySpark
    • Labels:
      None

      Description

      ML image can't work with numpy version prior to 1.9 now.

      Current pyspark test can show it:

      test_read_images (pyspark.ml.tests.test_image.ImageReaderTest) ... ERROR                                                                              
      test_read_images_multiple_times (pyspark.ml.tests.test_image.ImageReaderTest2) ... ok                                                                 
                                                                                                                                                            
      ======================================================================                                                                                
      ERROR: test_read_images (pyspark.ml.tests.test_image.ImageReaderTest)                                                                                 
      ----------------------------------------------------------------------                                                                                
      Traceback (most recent call last):
        File "/Users/viirya/docker_tmp/repos/spark-1/python/pyspark/ml/tests/test_image.py", line 36, in test_read_images                                   
          self.assertEqual(ImageSchema.toImage(array, origin=first_row[0]), first_row)                                                                      
        File "/Users/viirya/docker_tmp/repos/spark-1/python/pyspark/ml/image.py", line 193, in toImage                                                      
          data = bytearray(array.astype(dtype=np.uint8).ravel().tobytes())                                                                                  
      AttributeError: 'numpy.ndarray' object has no attribute 'tobytes'                                                                                     
                                                                                                                                                            
      ----------------------------------------------------------------------
      Ran 2 tests in 29.040s                                                                                                                                
                                                                                                                                                            
      FAILED (errors=1)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                viirya Liang-Chi Hsieh
                Reporter:
                viirya Liang-Chi Hsieh
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: