Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Write dates/timestamps to Avro file in Spark 2.4.5:

      $ export TZ="America/Los_Angeles"
      $ bin/spark-shell --packages org.apache.spark:spark-avro_2.11:2.4.5
      
      scala> df.write.format("avro").save("/Users/maxim/tmp/before_1582/2_4_5_ts_avro")
      
      scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_ts_avro").show(false)
      +----------+
      |date      |
      +----------+
      |1001-01-01|
      +----------+
      
      
      scala> df2.write.format("avro").save("/Users/maxim/tmp/before_1582/2_4_5_ts_avro")
      
      scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_ts_avro").show(false)
      +--------------------------+
      |ts                        |
      +--------------------------+
      |1001-01-01 01:02:03.123456|
      +--------------------------+
      

      Spark 3.0.0-preview2 ( and 3.1.0-SNAPSHOT) outputs different values from Spark 2.4.5:

      $ export TZ="America/Los_Angeles"
      $ /bin/spark-shell --packages org.apache.spark:spark-avro_2.12:2.4.5
      
      scala> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
      
      scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_date_avro").show(false)
      +----------+
      |date      |
      +----------+
      |1001-01-07|
      +----------+
      
      scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_ts_avro").show(false)
      +--------------------------+
      |ts                        |
      +--------------------------+
      |1001-01-07 01:09:05.123456|
      +--------------------------+
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maxgekk Maxim Gekk
                Reporter:
                maxgekk Maxim Gekk
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: