Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24768 Have a built-in AVRO data source implementation
  3. SPARK-25883

Override method `prettyName` in `from_avro`/`to_avro`

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 2.4.1, 3.0.0
    • SQL
    • None

    Description

      Previously in from_avro/to_avro, we override the method `simpleString` and `sql` for the string output. However, the override only affects the alias naming:
      ```
      Project [from_avro('col,
      ...
      , (mode,PERMISSIVE)) AS from_avro(col, struct<col1:bigint,col2:double>, Map(mode -> PERMISSIVE))#11]
      ```
      It only makes the alias name quite long.

      We should follow `from_csv`/`from_json` here, to override the method prettyName only, and we will get a clean alias name

      ```
      ... AS from_avro(col)#11
      ```

      Attachments

        Issue Links

          Activity

            People

              Gengliang.Wang Gengliang Wang
              Gengliang.Wang Gengliang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: