[SPARK-5509] EqualTo operator doesn't handle binary type properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0
• Fix Version/s: 1.3.0
• Component/s: SQL
• Target Version/s: 1.4.0
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 31/Jan/15 22:46
• Updated: 12/Feb/15 04:43
• Resolved: 05/Feb/15 23:30

Description

Binary type is mapped to Array[Byte], which can't be compared with == directly. However, EqualTo.eval() uses plain == to compare values. Run the following spark-shell snippet with Spark 1.2.0 to reproduce this issue:

import org.apache.spark.sql.SQLContext
import sc._

val sqlContext = new SQLContext(sc)
import sqlContext._

case class KV(key: Int, value: Array[Byte])

def toBinary(s: String): Array[Byte] = s.toString.getBytes("UTF-8")
registerFunction("toBinary", toBinary _)

parallelize(1 to 1024).map(i => KV(i, toBinary(i.toString))).registerTempTable("bin")

// OK
sql("select * from bin where value < toBinary('100')").collect()

// Oops, returns nothing
sql("select * from bin where value = toBinary('100')").collect()

Issue Links

• links to: [Github] Pull Request #4308 (liancheng)