[SPARK-26567] Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: eaton
• Created: 08/Jan/19 03:22
• Updated: 12/Dec/22 18:10
• Resolved: 08/Jan/19 03:26

Description

If we want to be consistent, we can modify the makeConverter function in UnivocityParser, but the performance may get worse.The modified code is as follows:

 

def makeConverter(
    name: String,
    dataType: DataType,
    nullable: Boolean = true,
    options: CSVOptions): ValueConverter = dataType match {
  case _: ByteType => (d: String) =>
    nullSafeDatum(d, name, nullable, options)(_.toDouble.intValue().toByte)

  case _: ShortType => (d: String) =>
    nullSafeDatum(d, name, nullable, options)(_.toDouble.intValue().toShort)

  case _: IntegerType => (d: String) =>
    nullSafeDatum(d, name, nullable, options)(_.toDouble.intValue())