Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41281 Feature parity: SparkSession API in Spark Connect
  3. SPARK-41745

SparkSession.createDataFrame does not respect the column names in the row

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Connect
    • None

    Description

      File "/.../spark/python/pyspark/sql/connect/group.py", line 173, in pyspark.sql.connect.group.GroupedData.pivot
      Failed example:
          df1.show()
      Differences (ndiff with -expected +actual):
          - +------+----+--------+
          ?                   ---
          + +------+----+-----+
          - |course|year|earnings|
          + |    _1|  _2|   _3|
          - +------+----+--------+
          ?                   ---
          + +------+----+-----+
          - |dotNET|2012|   10000|
          ?              ---
          + |dotNET|2012|10000|
          - |  Java|2012|   20000|
          ?              ---
          + |  Java|2012|20000|
          - |dotNET|2012|    5000|
          ?               ---
          + |dotNET|2012| 5000|
          - |dotNET|2013|   48000|
          ?              ---
          + |dotNET|2013|48000|
          - |  Java|2013|   30000|
          ?              ---
          + |  Java|2013|30000|
          - +------+----+--------+
          ?                   ---
          + +------+----+-----+
          + <BLANKLINE>
      

      Attachments

        Issue Links

          Activity

            People

              podongfeng Ruifeng Zheng
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: