Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39405

NumPy Input Support in PySpark

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Resolved
    • 3.4.0
    • None
    • PySpark
    • None

    Description

      NumPy is the fundamental package for scientific computing with Python. It is very commonly used, especially in the data science world. For example, Pandas is backed by NumPy, and Tensors also supports interchangeable conversion from/to NumPy arrays.

      However, PySpark only supports Python native types with the exception of pandas data in “SparkSession.createDataFrame(pandas.DataFrame)”. “DataFrame.toPandas()”. NumPy input support has been requested multiple times, see SPARK-2012, SPARK-37697, SPARK-31776, and SPARK-6857.

      The ticket proposes to support NumPy input in PySpark.
       
      See more design design.

      Attachments

        Issue Links

          Activity

            People

              XinrongM Xinrong Meng
              XinrongM Xinrong Meng
              hyukjin.kwon hyukjin.kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: