Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25638

Convert structs to CSV strings

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • SQL
    • None

    Description

      New function to_csv() should convert a column of StructType to a column with CSV string. The function should accept the CSV options as CSV data source does. API should expose the following methods:

      def to_csv(e: Column, options: Map[String, String]): Column
      

      and for Python/R:

      def to_csv(e: Column, options: java.util.Map[String, String]): Column
      

      Here is an example of usage:

      select to_csv(named_struct('time', to_timestamp('2015-08-26', 'yyyy-MM-dd')), map('timestampFormat', 'dd/MM/yyyy'));
      
      26/08/2015
      

      Attachments

        Activity

          People

            maxgekk Max Gekk
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: