Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36210

Preserve column insertion order in Dataset.withColumns

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.3, 3.1.2, 3.2.0
    • Fix Version/s: 3.2.0, 3.1.3, 3.0.4
    • Component/s: SQL
    • Labels:
      None

      Description

      Dataset.withColumns uses a Map (columnMap) to store the mapping of column name to column. however this loses the order of the columns. also none of the operations used on the Map (find and filter) benefits from the map's lookup features. so it seems simpler to use a Seq instead, which also preserves the insertion order.

        Attachments

          Activity

            People

            • Assignee:
              koert koert kuipers
              Reporter:
              koert koert kuipers
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: