[SPARK-38520] Overflow occurs when reading ANSI day time interval from CSV file

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: chong
• Created: 11/Mar/22 06:59
• Updated: 12/Dec/22 18:10

Description

Problem:

Overflow occurs when reading the following positive intervals, the results become to negative

interval '106751992' day     => INTERVAL '-106751990' DAY

INTERVAL +'[phone]' hour => INTERVAL '-2562047787' HOUR

interval '153722867281' minute => INTERVAL '-153722867280' MINUTE

 

Reproduce:

// days overflow
 scala> val schema = StructType(Seq(StructField("c1",
   DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))))
 scala> spark.read.csv(path).show(false)
 +------------------------+
 |_c0                     |
 +------------------------+
 |interval '106751992' day|
 +------------------------+
 scala> spark.read.schema(schema).csv(path).show(false)
 +-------------------------+
 |c1                       |
 +-------------------------+
 |INTERVAL '-106751990' DAY|
 +-------------------------+
  // hour overflow
 scala> val schema = StructType(Seq(StructField("c1",
   DayTimeIntervalType(DayTimeIntervalType.HOUR, DayTimeIntervalType.HOUR))))
 scala> spark.read.csv(path).show(false)
 +----------------------------+
 |_c0                         |
 +----------------------------+
 |INTERVAL +'[phone]' hour|
 +----------------------------+
 scala> spark.read.schema(schema).csv(path).show(false)
 +---------------------------+
 |c1                         |
 +---------------------------+
 |INTERVAL '-2562047787' HOUR|
 +---------------------------+
 // minute overflow
 scala> val schema = StructType(Seq(StructField("c1",
   DayTimeIntervalType(DayTimeIntervalType.MINUTE, DayTimeIntervalType.MINUTE))))
 scala> spark.read.csv(path).show(false)
 +------------------------------+
 |_c0                           |
 +------------------------------+
 |interval '153722867281' minute|
 +------------------------------+
 scala> spark.read.schema(schema).csv(path).show(false)
 +-------------------------------+
 |c1                             |
 +-------------------------------+
 |INTERVAL '-153722867280' MINUTE|
 +-------------------------------+

 

others:

Also check the negative value is read to positive.

 

others:

should check the negative also

Issue Links

• links to: [Github] Pull Request #35845 (res-life)