[ORC-1059] Align findColumns behaviour between 1.6 and 1.7 release

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0, 1.7.1
• Fix Version/s: 1.7.2
• Assignee: Panagiotis Garefalakis
• Reporter: Panagiotis Garefalakis
• Created: 13/Dec/21 12:35
• Updated: 02/Feb/22 23:55
• Resolved: 15/Dec/21 23:48

Description

ORC-741 changed the behaviour of column resolution throwing an IllegalArgumentException when a column is not found in the read schema.

Column resolution is also using by mapSargColumnsToOrcInternalColId when pushing down filters and throws an IllegalArgumentException exception when a filter column is not part of the schema instead of ignoring it (as it was the case in 1.6 and back).

1.6 and back

try {
  TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
      columnName, evolution.isSchemaEvolutionCaseAware);
  TypeDescription fileColumn = evolution.getFileType(readerColumn);
  return fileColumn == null ? -1 : fileColumn.getId();
} catch (IllegalArgumentException e) {
  if (LOG.isDebugEnabled()){
    LOG.debug("{}", e.getMessage());
  }
  return -1;
} 

1.7 and main

try {
  TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
    columnName, evolution.isSchemaEvolutionCaseAware);
  return evolution.getFileType(readerColumn);
} catch (IllegalArgumentException e) {
  throw new IllegalArgumentException("Filter could not find column with name: " +
                                     columnName + " on " + evolution.getReaderBaseSchema(),
                                     e);
} 

This makes harder to upgrade to 1.7 on downstream consumers like Hive https://issues.apache.org/jira/browse/HIVE-25497

Issue Links

• blocks: HIVE-25497 Bump ORC to 1.7.2
• is caused by: ORC-741 Schema Evolution missing column is not handled in the presence of filters
• links to: GitHub Pull Request #972