Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36210

Preserve column insertion order in Dataset.withColumns

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.3, 3.1.2, 3.2.0
    • 3.2.0, 3.1.3, 3.0.4
    • SQL
    • None

    Description

      Dataset.withColumns uses a Map (columnMap) to store the mapping of column name to column. however this loses the order of the columns. also none of the operations used on the Map (find and filter) benefits from the map's lookup features. so it seems simpler to use a Seq instead, which also preserves the insertion order.

      Attachments

        Activity

          People

            koert koert kuipers
            koert koert kuipers
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: