Hive
  1. Hive
  2. HIVE-2609

NPE when pruning partitions by thrift method get_partitions_by_filter

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.1
    • Fix Version/s: None
    • Component/s: Metastore
    • Labels:
      None

      Description

      It's a datanucleus bug indeed.

      try this code:

      boolean open = false;
      for (int i = 0; i < 5 && !open; ++i) {
        try {
          transport.open();
          open = true;
        } catch (TTransportException e) {
          System.out.println("failed to connect to MetaStore, re-trying...");
          try {
            Thread.sleep(1000);
          } catch (InterruptedException ignore) {}
        }
      }
      
      try {
        List<Partition> parts =
            client.get_partitions_by_filter("default", "partitioned_nation",
                "pt < '2'", (short) -1);
        for (Partition part : parts) {
          System.out.println(part.getSd().getLocation());
        }
      } catch (Exception te) {
        te.printStackTrace();
      }
      

      A NPEexception would be thrown on the thrift server side

      11/11/25 13:11:55 ERROR api.ThriftHiveMetastore$Processor: Internal error processing get_partitions_by_filter
      java.lang.NullPointerException
              at org.datanucleus.store.mapped.mapping.MappingHelper.getMappingIndices(MappingHelper.java:35)
              at org.datanucleus.store.mapped.expression.StatementText.applyParametersToStatement(StatementText.java:194)
              at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:233)
              at org.datanucleus.store.rdbms.query.legacy.SQLEvaluator.evaluate(SQLEvaluator.java:115)
              at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.performExecute(JDOQLQuery.java:288)
              at org.datanucleus.store.query.Query.executeQuery(Query.java:1657)
              at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
              at org.datanucleus.store.query.Query.executeWithMap(Query.java:1526)
              at org.datanucleus.jdo.JDOQuery.executeWithMap(JDOQuery.java:334)
              at org.apache.hadoop.hive.metastore.ObjectStore.listMPartitionsByFilter(ObjectStore.java:1329)
              at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilter(ObjectStore.java:1241)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2369)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2366)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_filter(HiveMetaStore.java:2366)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_filter.process(ThriftHiveMetastore.j
      ava:6099)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor.process(ThriftHiveMetastore.java:4789)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$TLoggingProcessor.process(HiveMetaStore.java:3167)
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      

      A null JavaTypeMapping was passed into org.datanucleus.store.mapped.mapping.MappingHelper.(int initialPosition, JavaTypeMapping mapping), that caused NPE.

      After digged into the datanucleus source, I found that the null value was born in the constructor of org.datanucleus.store.mapped.expression.SubstringExpression. see

          /**
           * Constructs the substring
           * @param str the String Expression
           * @param begin The start position
           * @param end The end position expression
           **/   
          public SubstringExpression(StringExpression str, NumericExpression begin, NumericExpression end)
          {
              super(str.getQueryExpression());
      
              st.append("SUBSTRING(").append(str).append(" FROM ")
                  .append(begin.add(new IntegerLiteral(qs, mapping, BigInteger.ONE)))
                  .append(" FOR ").append(end.sub(begin)).append(')');
          }
      

      The field mapping hasn't been instanced at that moment.

      How do you deal with such a external bug?

        Issue Links

          Activity

          Hide
          Mithun Radhakrishnan added a comment -

          I haven't done a whole bunch of testing yet, but I did try to replace the following libraries:
          1. datanucleus-core-2.0.3.jar -> datanucleus-core-2.2.5.jar
          2. datanucleus-rdbms-2.0.3.jar -> datanucleus-rdbms-2.2.4.jar

          This has met with a modicum of success, in that code such as "pt < '2'" does indeed work. (Mind you, the testing I did was on HCatalog, not on Hive-proper, but it still applies.)

          I know this has been discussed before, but I'd like to bring this back to the table: Could we please consider having Hive-Metastore depend on a less dated set of JDO libraries?

          Show
          Mithun Radhakrishnan added a comment - I haven't done a whole bunch of testing yet, but I did try to replace the following libraries: 1. datanucleus-core-2.0.3.jar -> datanucleus-core-2.2.5.jar 2. datanucleus-rdbms-2.0.3.jar -> datanucleus-rdbms-2.2.4.jar This has met with a modicum of success, in that code such as "pt < '2'" does indeed work. (Mind you, the testing I did was on HCatalog, not on Hive-proper, but it still applies.) I know this has been discussed before, but I'd like to bring this back to the table: Could we please consider having Hive-Metastore depend on a less dated set of JDO libraries?
          Hide
          Ashutosh Chauhan added a comment -

          @Mithun,
          Good finding. We can definitely upgrade the jdo libs. If all our tests pass, then certainly it makes sense to do so, given that there is atleast one bug (this jira) which will be fixed by it. There is some work for it already going on at HIVE-2084 for this. But, looks like Carl hit failures there. Also, if we are upgrading we should try to go to 3.0.1 instead of 2.x series.

          Show
          Ashutosh Chauhan added a comment - @Mithun, Good finding. We can definitely upgrade the jdo libs. If all our tests pass, then certainly it makes sense to do so, given that there is atleast one bug (this jira) which will be fixed by it. There is some work for it already going on at HIVE-2084 for this. But, looks like Carl hit failures there. Also, if we are upgrading we should try to go to 3.0.1 instead of 2.x series.
          Hide
          Aniket Mokashi added a comment -

          @Mithun,

          How did you change these jars? did you replace them after you compiled or did you replace them before build time?

          If I change the ivy cache, here is what I get-[datanucleusenhancer] Exception in thread "main" java.lang.NoSuchMethodError: org.datanucleus.metadata.AbstractMemberMetaData.getJdoFieldFlag()B
          [datanucleusenhancer] at org.datanucleus.enhancer.asm.JdoMethodAdapter.visitFieldInsn(JdoMethodAdapter.java:104)
          [datanucleusenhancer] at org.objectweb.asm.ClassReader.accept(Unknown Source)
          [datanucleusenhancer] at org.objectweb.asm.ClassReader.accept(Unknown Source)
          [datanucleusenhancer] at org.datanucleus.enhancer.asm.ASMClassEnhancer.enhance(ASMClassEnhancer.java:388)
          [datanucleusenhancer] at org.datanucleus.enhancer.DataNucleusEnhancer.enhanceClass(DataNucleusEnhancer.java:1035)
          [datanucleusenhancer] at org.datanucleus.enhancer.DataNucleusEnhancer.enhance(DataNucleusEnhancer.java:609)
          [datanucleusenhancer] at org.datanucleus.enhancer.DataNucleusEnhancer.main(DataNucleusEnhancer.java:1316)

          Show
          Aniket Mokashi added a comment - @Mithun, How did you change these jars? did you replace them after you compiled or did you replace them before build time? If I change the ivy cache, here is what I get- [datanucleusenhancer] Exception in thread "main" java.lang.NoSuchMethodError: org.datanucleus.metadata.AbstractMemberMetaData.getJdoFieldFlag()B [datanucleusenhancer] at org.datanucleus.enhancer.asm.JdoMethodAdapter.visitFieldInsn(JdoMethodAdapter.java:104) [datanucleusenhancer] at org.objectweb.asm.ClassReader.accept(Unknown Source) [datanucleusenhancer] at org.objectweb.asm.ClassReader.accept(Unknown Source) [datanucleusenhancer] at org.datanucleus.enhancer.asm.ASMClassEnhancer.enhance(ASMClassEnhancer.java:388) [datanucleusenhancer] at org.datanucleus.enhancer.DataNucleusEnhancer.enhanceClass(DataNucleusEnhancer.java:1035) [datanucleusenhancer] at org.datanucleus.enhancer.DataNucleusEnhancer.enhance(DataNucleusEnhancer.java:609) [datanucleusenhancer] at org.datanucleus.enhancer.DataNucleusEnhancer.main(DataNucleusEnhancer.java:1316)
          Hide
          Thomas Weise added a comment -

          We replace them after the build.

          Show
          Thomas Weise added a comment - We replace them after the build.
          Hide
          Aniket Mokashi added a comment -

          Hmm, unfortunately it didnt work for me, I am not at hive0.9 yet, that might be one of the reason. Anyway, for those who want take jar injection path -http://www.datanucleus.org/servlet/jira/browse/NUCJDOQUERY-4

          Show
          Aniket Mokashi added a comment - Hmm, unfortunately it didnt work for me, I am not at hive0.9 yet, that might be one of the reason. Anyway, for those who want take jar injection path - http://www.datanucleus.org/servlet/jira/browse/NUCJDOQUERY-4
          Hide
          Travis Crawford added a comment -

          I ran into this today too and, in addition to updating the two jars Thomas mentioned, also had to update:

          https://github.com/apache/hive/blob/trunk/metastore/src/model/package.jdo#L49

          In our hive tables the column is named "COMMENT" - not "FCOMMENT". Without updating datanucleus things work fine, but this change is required when updating jars. I don't understand why the change in behavior yet though.

          Show
          Travis Crawford added a comment - I ran into this today too and, in addition to updating the two jars Thomas mentioned, also had to update: https://github.com/apache/hive/blob/trunk/metastore/src/model/package.jdo#L49 In our hive tables the column is named "COMMENT" - not "FCOMMENT". Without updating datanucleus things work fine, but this change is required when updating jars. I don't understand why the change in behavior yet though.
          Hide
          Brock Noland added a comment -

          I tried reproducing this after HIVE-3632 and cannot. Therefore I think we can mark it resolved.

          Show
          Brock Noland added a comment - I tried reproducing this after HIVE-3632 and cannot. Therefore I think we can mark it resolved.

            People

            • Assignee:
              Unassigned
              Reporter:
              Min Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development