[SPARK-16152] `In` predicate does not work with null values

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.6.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Ashar Fuadi
• Created: 22/Jun/16 18:51
• Updated: 19/Aug/16 17:21
• Resolved: 19/Aug/16 17:21

Description

According to https://github.com/apache/spark/blob/v1.6.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L134..L136:

 override def eval(input: InternalRow): Any = {
    val evaluatedValue = value.eval(input)
    if (evaluatedValue == null) {
      null
    } else {
      ...

we always return null when the current value is null, ignoring the elements of list. Therefore, we cannot have a predicate which tests whether a column contains values in e.g. [1, 2, 3, null]

Is this a bug, or is this actually the expected behavior?