Hive
  1. Hive
  2. HIVE-5364

NPE on some queries from partitioned orc table

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      If you create a partitioned ORC table with:

      create table A
      ...
      PARTITIONED BY (
      year int,
      month int,
      day int)
      

      This query will fail:
      select count from A where where year=2013 and month=9 and day=15;

      1. D13215.1.patch
        31 kB
        Phabricator

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          This only happens in Hive trunk.

          Show
          Owen O'Malley added a comment - This only happens in Hive trunk.
          Hide
          Sergey Shelukhin added a comment -

          some error message would be useful

          Show
          Sergey Shelukhin added a comment - some error message would be useful
          Hide
          Owen O'Malley added a comment -

          Sorry, Sergey. It is an NPE.

          Diagnostic Messages for this Task:
          Error: java.io.IOException: java.lang.reflect.InvocationTargetException
          at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
          at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:343)
          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:290)
          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:404)
          at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:544)
          at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:167)
          at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
          Caused by: java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:329)
          ... 11 more
          Caused by: java.lang.NullPointerException
          at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:91)
          at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:203)
          at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
          ... 16 more
          
          Show
          Owen O'Malley added a comment - Sorry, Sergey. It is an NPE. Diagnostic Messages for this Task: Error: java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:343) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:290) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:404) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:544) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:167) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:329) ... 11 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:91) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:203) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65) ... 16 more
          Hide
          Phabricator added a comment -

          omalley requested code review of "HIVE-5364 [jira] NPE on some queries from partitioned orc table".

          Reviewers: JIRA

          HIVE-5364 NPE in OrcInputFormat

          If you create a partitioned ORC table with:

          create table A (
          id int
          ) PARTITIONED BY (
          year int,
          month int,
          day int)

          This query will fail when predicate pushdown is on:
          SELECT id FROM A WHERE id > 90;

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D13215

          AFFECTED FILES
          data/files/orc_create_people.txt
          ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
          ql/src/test/queries/clientpositive/orc_create.q
          ql/src/test/results/clientpositive/orc_create.q.out

          MANAGE HERALD RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/38853/

          To: JIRA, omalley

          Show
          Phabricator added a comment - omalley requested code review of " HIVE-5364 [jira] NPE on some queries from partitioned orc table". Reviewers: JIRA HIVE-5364 NPE in OrcInputFormat If you create a partitioned ORC table with: create table A ( id int ) PARTITIONED BY ( year int, month int, day int) This query will fail when predicate pushdown is on: SELECT id FROM A WHERE id > 90; TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D13215 AFFECTED FILES data/files/orc_create_people.txt ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/test/queries/clientpositive/orc_create.q ql/src/test/results/clientpositive/orc_create.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/38853/ To: JIRA, omalley
          Hide
          Owen O'Malley added a comment -

          This patch fixes both problems.

          Show
          Owen O'Malley added a comment - This patch fixes both problems.
          Hide
          Gunther Hagleitner added a comment -

          Owen O'Malley: Looks good, except that I think VectorizedOrcInputFormat needs to be updated as well.

          Show
          Gunther Hagleitner added a comment - Owen O'Malley : Looks good, except that I think VectorizedOrcInputFormat needs to be updated as well.
          Hide
          Owen O'Malley added a comment -

          I don't think the vectorized input has merged with the predicate push down yet, so the relevant code isn't there yet.

          Show
          Owen O'Malley added a comment - I don't think the vectorized input has merged with the predicate push down yet, so the relevant code isn't there yet.
          Hide
          Gunther Hagleitner added a comment -

          Sorry, my bad.

          Show
          Gunther Hagleitner added a comment - Sorry, my bad.
          Hide
          Phabricator added a comment -

          hagleitn has commented on the revision "HIVE-5364 [jira] NPE on some queries from partitioned orc table".

          LGTM +1

          REVISION DETAIL
          https://reviews.facebook.net/D13215

          To: JIRA, omalley
          Cc: hagleitn

          Show
          Phabricator added a comment - hagleitn has commented on the revision " HIVE-5364 [jira] NPE on some queries from partitioned orc table". LGTM +1 REVISION DETAIL https://reviews.facebook.net/D13215 To: JIRA, omalley Cc: hagleitn
          Hide
          Thejas M Nair added a comment -

          I checked with Owen, this is an issue in hive 0.12 as well.

          Show
          Thejas M Nair added a comment - I checked with Owen, this is an issue in hive 0.12 as well.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12606384/D13215.1.patch

          SUCCESS: +1 4051 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1027/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1027/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12606384/D13215.1.patch SUCCESS: +1 4051 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1027/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1027/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Thejas M Nair added a comment -

          Patch committed to 0.12 branch and trunk.
          Thanks for the contribution Owen, and thanks for the review Gunther!

          Show
          Thejas M Nair added a comment - Patch committed to 0.12 branch and trunk. Thanks for the contribution Owen, and thanks for the review Gunther!
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-branch-0.12-hadoop1 #10 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop1/10/)
          HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529098)

          • /hive/branches/branch-0.12/data/files/orc_create_people.txt
          • /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
          • /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
          • /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_create.q
          • /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_create.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-branch-0.12-hadoop1 #10 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop1/10/ ) HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529098 ) /hive/branches/branch-0.12/data/files/orc_create_people.txt /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_create.q /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_create.q.out
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #124 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/124/)
          HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097)

          • /hive/trunk/data/files/orc_create_people.txt
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
          • /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q
          • /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #124 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/124/ ) HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097 ) /hive/trunk/data/files/orc_create_people.txt /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2 #475 (See https://builds.apache.org/job/Hive-trunk-hadoop2/475/)
          HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097)

          • /hive/trunk/data/files/orc_create_people.txt
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
          • /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q
          • /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #475 (See https://builds.apache.org/job/Hive-trunk-hadoop2/475/ ) HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097 ) /hive/trunk/data/files/orc_create_people.txt /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2379 (See https://builds.apache.org/job/Hive-trunk-h0.21/2379/)
          HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097)

          • /hive/trunk/data/files/orc_create_people.txt
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
          • /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q
          • /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2379 (See https://builds.apache.org/job/Hive-trunk-h0.21/2379/ ) HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097 ) /hive/trunk/data/files/orc_create_people.txt /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop1-ptest #190 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/190/)
          HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097)

          • /hive/trunk/data/files/orc_create_people.txt
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
          • /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q
          • /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #190 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/190/ ) HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529097 ) /hive/trunk/data/files/orc_create_people.txt /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java /hive/trunk/ql/src/test/queries/clientpositive/orc_create.q /hive/trunk/ql/src/test/results/clientpositive/orc_create.q.out
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-branch-0.12-hadoop2 #6 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop2/6/)
          HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529098)

          • /hive/branches/branch-0.12/data/files/orc_create_people.txt
          • /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
          • /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
          • /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_create.q
          • /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_create.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-branch-0.12-hadoop2 #6 (See https://builds.apache.org/job/Hive-branch-0.12-hadoop2/6/ ) HIVE-5364 : NPE on some queries from partitioned orc table (Owen O'Malley via Gunther Hagleitner) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529098 ) /hive/branches/branch-0.12/data/files/orc_create_people.txt /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java /hive/branches/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java /hive/branches/branch-0.12/ql/src/test/queries/clientpositive/orc_create.q /hive/branches/branch-0.12/ql/src/test/results/clientpositive/orc_create.q.out
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development