[LIVY-601] Add support for user defined date format and timezone when transform spark result rows to json

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Fix Version/s: 0.9.0
• Component/s: REPL
• Assignee: Unassigned
• Reporter: iteblog
• Created: 27/Jun/19 06:52
• Updated: 12/Nov/22 22:10

Description

When Livy get the row data from Spark DataFrame and transform those rows to json, we use DefaultFormats to transform Spark timestamp to scala timestamp which use default timezone.

private implicit def formats = DefaultFormats

val result = spark.sql(code)
val schema = parse(result.schema.json)

// Get the row data
val rows = result.take(maxResult)
  .map {
    _.toSeq.map {
      // Convert java BigDecimal type to Scala BigDecimal, because current version of
      // Json4s doesn't support java BigDecimal as a primitive type (LIVY-455).
      case i: java.math.BigDecimal => BigDecimal(i)
      case e => e
    }
  }

val jRows = Extraction.decompose(rows)

// DefaultFormats defined in org.json4s.DefaultFormats
/** Default date format is UTC time.
 */
object DefaultFormats extends DefaultFormats {
  val UTC = TimeZone.getTimeZone("UTC")

  val losslessDate = {
    def createSdf = {
      val f = new java.text.SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
      f.setTimeZone(UTC)
      f
    }
    new ThreadLocal(createSdf)
  }


}

It would be useful if we can customize the time zone and time format