Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40527

Keep struct field names or map keys in CreateStruct

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None

    Description

      Using index-like notation when extracting columns in a struct produces generated column names like col1, col2, etc. However, when dot-like syntax is used, the column name is correctly selected. 

       

      Example:

      select struct(a['x'], a['y']), struct(a.x, a.y) from (select named_struct('x', 1, 'y', 2) as a) 

      Produces the following schema and result:

      root
       |-- struct(a.x, a.y): struct (nullable = false)
       |    |-- col1: integer (nullable = false)
       |    |-- col2: integer (nullable = false)
       |-- struct(a.x, a.y): struct (nullable = false)
       |    |-- x: integer (nullable = false)
       |    |-- y: integer (nullable = false)
      
      +----------------+----------------+
      |struct(a.x, a.y)|struct(a.x, a.y)|
      +----------------+----------------+
      |{1, 2}          |{1, 2}          |
      +----------------+----------------+ 

       

      It would be good to have x and y column names in the first struct.

      Attachments

        Activity

          People

            ivan.sadikov Ivan Sadikov
            ivan.sadikov Ivan Sadikov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: