Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27790 Support ANSI SQL INTERVAL types
  3. SPARK-35116

The generated data fits the precision of DayTimeIntervalType in spark

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • SQL
    • None

    Description

      The precision of java.time.Duration is nanosecond, but when it is used as DayTimeIntervalType in Spark, it is microsecond.
      At present, the DayTimeIntervalType data generated in the implementation of RandomDataGenerator is accurate to nanosecond, which will cause the DayTimeIntervalType to be converted to long, and then back to DayTimeIntervalType to lose the accuracy, which will cause the test to fail. For example: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137390/testReport/org.apache.spark.sql.hive.execution/HashAggregationQueryWithControlledFallbackSuite/udaf_with_all_data_types/

      Attachments

        Activity

          People

            beliefer Jiaan Geng
            beliefer Jiaan Geng
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: