[PARQUET-173] StatisticsFilter doesn't handle And properly

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.0
• Component/s: parquet-mr
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 28/Jan/15 09:09
• Updated: 13/Aug/15 08:23
• Resolved: 03/Feb/15 20:54

Description

I guess it's a pretty straightforward mistake

  @Override
  public Boolean visit(And and) {
    return and.getLeft().accept(this) && and.getRight().accept(this);
  }

  @Override
  public Boolean visit(Or or) {
    // seems unintuitive to put an && not an || here
    // but we can only drop a chunk of records if we know that
    // both the left and right predicates agree that no matter what
    // we don't need this chunk.
    return or.getLeft().accept(this) && or.getRight().accept(this);
  }

The consequence is that filter predicates like a > 10 && a < 20 can never drop any row groups.

Issue Links

• blocks: SPARK-5451 And predicates are not properly pushed down
• is duplicated by: PARQUET-250 StatisticsFilter "And"'s semantic error