Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11806 Spark 2.0 deprecations and removals
  3. SPARK-13594

remove typed operations (map, flatMap, mapPartitions) from Python DataFrame

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Once we implement Dataset-equivalent API in Python, we'd need to change the return type of map, flatMap, and mapPartitions. In this case, we should just remove them from Python DataFrame now in 2.0, so we don't break APIs in 2.x.

      Users can still use those after the removal, but must prefix "rdd" to it. For example, df.rdd.map, df.rdd.flatMap, and df.rdd.mapPartitions.

        Attachments

          Activity

            People

            • Assignee:
              cloud_fan Wenchen Fan
              Reporter:
              cloud_fan Wenchen Fan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: