Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-601

Add support for user defined date format and timezone when transform spark result rows to json

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: REPL
    • Labels:
      None

      Description

      When Livy get the row data from Spark DataFrame and transform those rows to json, we use DefaultFormats to transform Spark timestamp to scala timestamp which use default timezone.

      private implicit def formats = DefaultFormats
      
      val result = spark.sql(code)
      val schema = parse(result.schema.json)
      
      // Get the row data
      val rows = result.take(maxResult)
        .map {
          _.toSeq.map {
            // Convert java BigDecimal type to Scala BigDecimal, because current version of
            // Json4s doesn't support java BigDecimal as a primitive type (LIVY-455).
            case i: java.math.BigDecimal => BigDecimal(i)
            case e => e
          }
        }
      
      val jRows = Extraction.decompose(rows)
      
      // DefaultFormats defined in org.json4s.DefaultFormats
      /** Default date format is UTC time.
       */
      object DefaultFormats extends DefaultFormats {
        val UTC = TimeZone.getTimeZone("UTC")
      
        val losslessDate = {
          def createSdf = {
            val f = new java.text.SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
            f.setTimeZone(UTC)
            f
          }
          new ThreadLocal(createSdf)
        }
      
      
      }

      It would be useful if we can customize the time zone and time format

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              397090770 wyp
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: