[SPARK-26551] Selecting one complex field and having is null predicate on another complex field can cause error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: SQL
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 06/Jan/19 03:58
• Updated: 11/Jan/19 19:26
• Resolved: 11/Jan/19 19:24

Description

The query below can cause error when doing schema pruning:

val query = sql("select * from contacts")
  .where("name.middle is not null")
  .select(
    "id",
    "name.first",
    "name.middle",
    "name.last"
  )
  .where("last = 'Jones'")
  .select(count("id")

Issue Links

• links to: GitHub Pull Request #23474