|
|
|
SPARK-47494
|
SPARK-35662
Add migration doc for the behavior change of Parquet timestamp inference since Spark 3.3
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-47493
|
SPARK-35662
Disable spark.sql.parquet.inferTimestampNTZ.enabled by default
|
Gengliang Wang
|
Gengliang Wang
|
|
Open |
Unresolved
|
|
|
|
|
|
|
|
SPARK-47447
|
SPARK-35662
Allow reading Parquet TimestampLTZ as TimestampNTZ
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-47370
|
SPARK-35662
Add migration doc: TimestampNTZ type inference on Parquet files
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-47368
|
SPARK-35662
Remove inferTimestampNTZ config check in ParquetRowConverter
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-43336
|
SPARK-35662
Casting between Timestamp and TimestampNTZ requires timezone
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-43072
|
SPARK-35662
Include TIMESTAMP_NTZ in ANSI Compliance doc
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42796
|
SPARK-35662
Support TimestampNTZ in Cached Batch
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42777
|
SPARK-35662
Support converting TimestampNTZ catalog stats to plan stats
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42743
|
SPARK-35662
Support analyze TimestampNTZ columns
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42723
|
SPARK-35662
Support parser data type json "timestamp_ltz" as TimestampType
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42455
|
SPARK-35662
Rename JDBC option inferTimestampNTZType as preferTimestampNTZ
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42442
|
SPARK-35662
Use spark.sql.timestampType for data source inference
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42430
|
SPARK-35662
Add documentation for TimestampNTZ type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42345
|
SPARK-35662
Rename TimestampNTZ inference conf as spark.sql.sources.timestampNTZTypeInference.enabled
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42296
|
SPARK-35662
Apply spark.sql.inferTimestampNTZInDataSources.enabled on JDBC data source
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42285
|
SPARK-35662
Introduce conf spark.sql.parquet.inferTimestampNTZ.enabled for TimestampNTZ inference on Parquet
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42243
|
SPARK-35662
Use `spark.sql.inferTimestampNTZInDataSources.enabled` to infer timestamp type on partition columns
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-42221
|
SPARK-35662
Introduce a new conf for TimestampNTZ schema inference in JSON/CSV
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-41462
|
SPARK-35662
Date and timestamp type can up cast to TimestampNTZ
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-39339
|
SPARK-35662
Support TimestampNTZ in JDBC data source
|
Ivan Sadikov
|
Ivan Sadikov
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-38984
|
SPARK-35662
Allow comparison between TimestampNTZ and Timestamp/Date
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-38834
|
SPARK-35662
Update the version of TimestampNTZ related changes as 3.4.0
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-38829
|
SPARK-35662
New configuration for controlling timestamp inference of Parquet
|
Ivan Sadikov
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-38828
|
SPARK-35662
Remove TimestampNTZ type Python support in Spark 3.3
|
Haejoon Lee
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-38813
|
SPARK-35662
Remove TimestampNTZ type support in Spark 3.3
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-38504
|
SPARK-35662
Can't read TimestampNTZ as TimestampLTZ
|
Jiaan Geng
|
Jiaan Geng
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-37990
|
SPARK-35662
Support TimestampNTZ in RowToColumnConverter
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-37986
|
SPARK-35662
Support TimestampNTZ radix sort
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-37707
|
SPARK-35662
Allow store assignment between TimestampNTZ and Date/Timestamp
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-37463
|
SPARK-35662
Read/Write Timestamp ntz from/to Orc uses int64
|
Jiaan Geng
|
Jiaan Geng
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-37385
|
SPARK-35662
Add tests for TimestampNTZ and TimestampLTZ for Parquet data source
|
Ivan Sadikov
|
Ivan Sadikov
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-37360
|
SPARK-35662
Support TimestampNTZ in JSON data source
|
Ivan Sadikov
|
Ivan Sadikov
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-37326
|
SPARK-35662
Support TimestampNTZ in CSV data source
|
Ivan Sadikov
|
Ivan Sadikov
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36724
|
SPARK-35662
Support timestamp_ntz as a type of time column for SessionWindow
|
Kousuke Saruta
|
Kousuke Saruta
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36675
|
SPARK-35662
Support ScriptTransformation for timestamp_ntz
|
Kousuke Saruta
|
Kousuke Saruta
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36661
|
SPARK-35662
Support TimestampNTZ in Py4J
|
Hyukjin Kwon
|
Hyukjin Kwon
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36626
|
SPARK-35662
Support TimestampNTZ in createDataFrame/toPandas and Python UDFs
|
Hyukjin Kwon
|
Hyukjin Kwon
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36625
|
SPARK-35662
Support TimestampNTZ in pandas API on Spark
|
Hyukjin Kwon
|
Hyukjin Kwon
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36608
|
SPARK-35662
Support TimestampNTZ in Arrow
|
Hyukjin Kwon
|
Hyukjin Kwon
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36491
|
SPARK-35662
Make from_json/to_json to handle timestamp_ntz type properly
|
Kousuke Saruta
|
Kousuke Saruta
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36490
|
SPARK-35662
Make from_csv/to_csv to handle timestamp_ntz type properly
|
Kousuke Saruta
|
Kousuke Saruta
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36409
|
SPARK-35662
Splitting test cases from datetime.sql
|
Wenchen Fan
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36357
|
SPARK-35662
Support pushdown Timestamp with local time zone for orc
|
Jiaan Geng
|
Jiaan Geng
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36346
|
SPARK-35662
Support TimestampNTZ type in Orc file source
|
Jiaan Geng
|
Jiaan Geng
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36257
|
SPARK-35662
Updated the version of TimestampNTZ related changes as 3.3.0
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36227
|
SPARK-35662
Remove TimestampNTZ type support in Spark 3.2
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36182
|
SPARK-35662
Support TimestampNTZ type in Parquet file source
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36180
|
SPARK-35662
Support TimestampNTZ type in Hive
|
Unassigned
|
Kent Yao 2
|
|
Resolved |
Won't Fix
|
|
|
|
|
|
|
|
SPARK-36179
|
SPARK-35662
Support TimestampNTZType in SparkGetColumnsOperation
|
Kent Yao 2
|
Kent Yao 2
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36175
|
SPARK-35662
Support TimestampNTZ in Avro data source
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36135
|
SPARK-35662
Support TimestampNTZ type in file partitioning
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36120
|
SPARK-35662
Support TimestampNTZ type in cache table
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36119
|
SPARK-35662
Add new SQL function to_timestamp_ltz
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36111
|
SPARK-35662
Support casting of timestamp without time zone to numeric type
|
Unassigned
|
Jiaan Geng
|
|
Resolved |
Won't Fix
|
|
|
|
|
|
|
|
SPARK-36091
|
SPARK-35662
Support TimestampNTZ type in expression TimeWindow
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36090
|
SPARK-35662
Support TimestampNTZType in expression Sequence
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36083
|
SPARK-35662
make_timestamp: return different result based on the default timestamp type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36072
|
SPARK-35662
TO_TIMESTAMP: return different results based on the default timestamp type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36055
|
SPARK-35662
Assign pretty SQL string to TimestampNTZ literals
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36054
|
SPARK-35662
Support group by TimestampNTZ column
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36050
|
SPARK-35662
Spark doesn’t support reading/writing TIMESTAMP_NTZ with ORC
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Not A Problem
|
|
|
|
|
|
|
|
SPARK-36046
|
SPARK-35662
Support new functions make_timestamp_ntz and make_timestamp_ltz
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36045
|
SPARK-35662
TO_UTC_TIMESTAMP and FROM_UTC_TIMESTAMP should return TimestampNTZ
|
Unassigned
|
Gengliang Wang
|
|
Resolved |
Won't Do
|
|
|
|
|
|
|
|
SPARK-36044
|
SPARK-35662
Suport TimestampNTZ in functions unix_timestamp/to_unix_timestamp
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36043
|
SPARK-35662
Add end-to-end tests with default timestamp type as TIMESTAMP_NTZ
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36037
|
SPARK-35662
Support ANSI SQL LOCALTIMESTAMP datetime value function
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36017
|
SPARK-35662
Support TimestampNTZType in expression ApproximatePercentile
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36016
|
SPARK-35662
Support TimestampNTZType in expression ApproxCountDistinctForIntervals
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-36015
|
SPARK-35662
Support TimestampNTZType in the Window spec definition
|
Jiaan Geng
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35979
|
SPARK-35662
Return different timestamp literals based on the default timestamp type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35978
|
SPARK-35662
Support non-reserved keyword TIMESTAMP_LTZ
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35977
|
SPARK-35662
Support non-reserved keyword TIMESTAMP_NTZ
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35975
|
SPARK-35662
New configuration spark.sql.timestampType for the default timestamp type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35971
|
SPARK-35662
Rename the type name of TimestampNTZType as "timestamp_ntz"
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35963
|
SPARK-35662
Rename TimestampWithoutTZType to TimestampNTZType
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35953
|
SPARK-35662
Support extracting date fields from timestamp without time zone
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35932
|
SPARK-35662
Support extracting hour/minute/second from timestamp without time zone
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35927
|
SPARK-35662
Remove type collection AllTimestampTypes
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35916
|
SPARK-35662
Support subtraction among Date/Timestamp/TimestampWithoutTZ
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35895
|
SPARK-35662
Support subtracting Intervals from TimestampWithoutTZ
|
Apache Spark
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35889
|
SPARK-35662
Support adding TimestampWithoutTZ with Interval types
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35854
|
SPARK-35662
Improve the error message of to_timestamp_ntz with invalid format pattern
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35839
|
SPARK-35662
New SQL function: to_timestamp_ntz
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35764
|
SPARK-35662
Assign pretty names to TimestampWithoutTZType
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35720
|
SPARK-35662
Support casting of String to timestamp without time zone type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35719
|
SPARK-35662
Support type conversion between timestamp and timestamp without time zone type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35718
|
SPARK-35662
Support casting of Date to timestamp without time zone type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35716
|
SPARK-35662
Support casting of timestamp without time zone to date type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35711
|
SPARK-35662
Support casting of timestamp without time zone to timestamp type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35698
|
SPARK-35662
Support casting of timestamp without time zone to strings
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35697
|
SPARK-35662
Test TimestampWithoutTZType as ordered and atomic type
|
Apache Spark
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35674
|
SPARK-35662
Test timestamp without time zone in UDF
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35664
|
SPARK-35662
Support java.time. LocalDateTime as an external type of TimestampWithoutTZ type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
SPARK-35663
|
SPARK-35662
Add Timestamp without time zone type
|
Gengliang Wang
|
Gengliang Wang
|
|
Resolved |
Fixed
|
|
|
|
|