Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-1059

Align findColumns behaviour between 1.6 and 1.7 release

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.7.0, 1.7.1
    • 1.7.2
    • None
    • None

    Description

      ORC-741 changed the behaviour of column resolution throwing an IllegalArgumentException when a column is not found in the read schema.

      Column resolution is also using by mapSargColumnsToOrcInternalColId when pushing down filters and throws an IllegalArgumentException exception when a filter column is not part of the schema instead of ignoring it (as it was the case in 1.6 and back).

      1.6 and back

      try {
        TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
            columnName, evolution.isSchemaEvolutionCaseAware);
        TypeDescription fileColumn = evolution.getFileType(readerColumn);
        return fileColumn == null ? -1 : fileColumn.getId();
      } catch (IllegalArgumentException e) {
        if (LOG.isDebugEnabled()){
          LOG.debug("{}", e.getMessage());
        }
        return -1;
      } 

      1.7 and main

      try {
        TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
          columnName, evolution.isSchemaEvolutionCaseAware);
        return evolution.getFileType(readerColumn);
      } catch (IllegalArgumentException e) {
        throw new IllegalArgumentException("Filter could not find column with name: " +
                                           columnName + " on " + evolution.getReaderBaseSchema(),
                                           e);
      } 

      This makes harder to upgrade to 1.7 on downstream consumers like Hive https://issues.apache.org/jira/browse/HIVE-25497

      Attachments

        Issue Links

          Activity

            People

              pgaref Panagiotis Garefalakis
              pgaref Panagiotis Garefalakis
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: