Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25638

Convert structs to CSV strings

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      New function to_csv() should convert a column of StructType to a column with CSV string. The function should accept the CSV options as CSV data source does. API should expose the following methods:

      def to_csv(e: Column, options: Map[String, String]): Column
      

      and for Python/R:

      def to_csv(e: Column, options: java.util.Map[String, String]): Column
      

      Here is an example of usage:

      select to_csv(named_struct('time', to_timestamp('2015-08-26', 'yyyy-MM-dd')), map('timestampFormat', 'dd/MM/yyyy'));
      
      26/08/2015
      

        Attachments

          Activity

            People

            • Assignee:
              maxgekk Maxim Gekk
              Reporter:
              maxgekk Maxim Gekk
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: