Hive
  1. Hive
  2. HIVE-5601

NPE in ORC's PPD when using select * from table with where predicate

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0, 0.12.1
    • Component/s: None
    • Labels:

      Description

      ORCInputFormat has a method findIncludedColumns() which returns boolean array of included columns. In case of the following query

      select * from qlog_orc where id<1000 limit 10;

      where all columns are selected the findIncludedColumns() returns null. This will result in a NPE when PPD is enabled. Following is the stack trace

      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.planReadPartialDataStreams(RecordReaderImpl.java:2387)
      	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:2543)
      	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2200)
      	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2573)
      	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2615)
      	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:132)
      	at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:348)
      	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:99)
      	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:241)
      	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:237)
      	... 8 more
      1. HIVE-5601.5.patch.txt
        9 kB
        Prasanth Jayachandran
      2. HIVE-5601.4-branch-0.12.patch.txt
        10 kB
        Prasanth Jayachandran
      3. HIVE-5601.trunk.5.patch.txt
        9 kB
        Prasanth Jayachandran
      4. HIVE-5601.trunk.4.patch.txt
        9 kB
        Prasanth Jayachandran
      5. HIVE-5601.branch-0.12.4.patch.txt
        10 kB
        Prasanth Jayachandran
      6. HIVE-5601.branch-0.12.3.patch.txt
        9 kB
        Prasanth Jayachandran
      7. HIVE-5601.trunk.3.patch.txt
        6 kB
        Prasanth Jayachandran
      8. HIVE-5601.branch-0.12.2.patch.txt
        1 kB
        Prasanth Jayachandran
      9. HIVE-5601.trunk.2.patch.txt
        1 kB
        Prasanth Jayachandran
      10. HIVE-5601.trunk.1.patch.txt
        0.7 kB
        Prasanth Jayachandran
      11. HIVE-5601.branch-12.1.patch.txt
        0.7 kB
        Prasanth Jayachandran

        Activity

        Hide
        Owen O'Malley added a comment -

        Actually, can you fix it lower down and make the Reader.rows replace null with a boolean[] of the right length?

        Show
        Owen O'Malley added a comment - Actually, can you fix it lower down and make the Reader.rows replace null with a boolean[] of the right length?
        Hide
        Prasanth Jayachandran added a comment -

        Updated the patch as per Owen O'Malley's comment..

        Show
        Prasanth Jayachandran added a comment - Updated the patch as per Owen O'Malley 's comment..
        Hide
        Prasanth Jayachandran added a comment -

        added test cases and updated with a fix for failing test case in branch-0.12.

        Show
        Prasanth Jayachandran added a comment - added test cases and updated with a fix for failing test case in branch-0.12.
        Hide
        Owen O'Malley added a comment -

        The length of the array needs to be the length of the types array, so you need to use footer.getTypesCount(). You shouldn't protect it with a null check on columnNames.

        Show
        Owen O'Malley added a comment - The length of the array needs to be the length of the types array, so you need to use footer.getTypesCount(). You shouldn't protect it with a null check on columnNames.
        Hide
        Prasanth Jayachandran added a comment -

        Owen O'Malley updated the patch with types count.

        Show
        Prasanth Jayachandran added a comment - Owen O'Malley updated the patch with types count.
        Hide
        Prasanth Jayachandran added a comment -

        Refreshed the patch with mavenized trunk.

        Show
        Prasanth Jayachandran added a comment - Refreshed the patch with mavenized trunk.
        Hide
        Owen O'Malley added a comment -

        +1

        Show
        Owen O'Malley added a comment - +1
        Hide
        Prasanth Jayachandran added a comment -

        Re-uploading patch for HIVE QA to run tests.

        Show
        Prasanth Jayachandran added a comment - Re-uploading patch for HIVE QA to run tests.
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12612739/HIVE-5601.5.patch.txt

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/194/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/194/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Tests failed with: IllegalArgumentException: null
        

        This message is automatically generated.

        ATTACHMENT ID: 12612739

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12612739/HIVE-5601.5.patch.txt Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/194/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/194/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Tests failed with: IllegalArgumentException: null This message is automatically generated. ATTACHMENT ID: 12612739
        Hide
        Prasanth Jayachandran added a comment -

        Reuploading patches as HIVE QA failed executing tests in previous run.

        Show
        Prasanth Jayachandran added a comment - Reuploading patches as HIVE QA failed executing tests in previous run.
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12612770/HIVE-5601.5.patch.txt

        SUCCESS: +1 4566 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/202/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/202/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12612770

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12612770/HIVE-5601.5.patch.txt SUCCESS: +1 4566 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/202/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/202/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12612770
        Hide
        Gunther Hagleitner added a comment -

        Committed the patch to trunk. Haven't updated hive .12 yet.

        Show
        Gunther Hagleitner added a comment - Committed the patch to trunk. Haven't updated hive .12 yet.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-branch-0.12-hadoop1 #32 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop1/32/)
        HIVE-5601: NPE in ORC's PPD when using select * from table with where
        predicate pushdown (Prasanth J via Owen O'Malley and Gunther Hagleitner) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542024)

        • /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
        • /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_predicate_pushdown.q
        • /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-branch-0.12-hadoop1 #32 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop1/32/ ) HIVE-5601 : NPE in ORC's PPD when using select * from table with where predicate pushdown (Prasanth J via Owen O'Malley and Gunther Hagleitner) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542024 ) /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_predicate_pushdown.q /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-branch-0.12-hadoop2 #23 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop2/23/)
        HIVE-5601: NPE in ORC's PPD when using select * from table with where
        predicate pushdown (Prasanth J via Owen O'Malley and Gunther Hagleitner) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542024)

        • /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
        • /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_predicate_pushdown.q
        • /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-branch-0.12-hadoop2 #23 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop2/23/ ) HIVE-5601 : NPE in ORC's PPD when using select * from table with where predicate pushdown (Prasanth J via Owen O'Malley and Gunther Hagleitner) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542024 ) /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_predicate_pushdown.q /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out

          People

          • Assignee:
            Prasanth Jayachandran
            Reporter:
            Prasanth Jayachandran
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development