Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30781

Missing SortedMap type in pyspark

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.4.5
    • Fix Version/s: None
    • Component/s: PySpark
    • Labels:

      Description

      Currently there is only MapType in pyspark API, which does not keep order of key-value map.

       

      SortedMapType would fill this gap: a map with sorted key-value pairs (like TreeMap in Java).

       

      For example:

      SortedMapType would be very useful when user wants to persist row from DataFrame into Mongo-db.

      row which consist a column with values of an ordered map type:

      col : [1 -> 22, 2 -> 16, 3 -> 25]

      will be persisted as following json:

      {   "1": 22,   "2": 16,   "3": 25 }

       

      instead of MapType which currently results in:

      {   "2": 16,   "1": 22,   "3": 25 }

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              urbanq Mateusz User
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: