Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11806 Spark 2.0 deprecations and removals
  3. SPARK-13594

remove typed operations (map, flatMap, mapPartitions) from Python DataFrame

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • SQL
    • None

    Description

      Once we implement Dataset-equivalent API in Python, we'd need to change the return type of map, flatMap, and mapPartitions. In this case, we should just remove them from Python DataFrame now in 2.0, so we don't break APIs in 2.x.

      Users can still use those after the removal, but must prefix "rdd" to it. For example, df.rdd.map, df.rdd.flatMap, and df.rdd.mapPartitions.

      Attachments

        Activity

          People

            cloud_fan Wenchen Fan
            cloud_fan Wenchen Fan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: