[HIVE-2609] NPE when pruning partitions by thrift method get_partitions_by_filter

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.1
• Component/s: Metastore
• Assignee: Unassigned
• Reporter: Min Zhou
• Created: 25/Nov/11 08:26
• Updated: 13/Aug/13 15:20

Description

It's a datanucleus bug indeed.

try this code:

boolean open = false;
for (int i = 0; i < 5 && !open; ++i) {
  try {
    transport.open();
    open = true;
  } catch (TTransportException e) {
    System.out.println("failed to connect to MetaStore, re-trying...");
    try {
      Thread.sleep(1000);
    } catch (InterruptedException ignore) {}
  }
}

try {
  List<Partition> parts =
      client.get_partitions_by_filter("default", "partitioned_nation",
          "pt < '2'", (short) -1);
  for (Partition part : parts) {
    System.out.println(part.getSd().getLocation());
  }
} catch (Exception te) {
  te.printStackTrace();
}

A NPEexception would be thrown on the thrift server side

11/11/25 13:11:55 ERROR api.ThriftHiveMetastore$Processor: Internal error processing get_partitions_by_filter
java.lang.NullPointerException
        at org.datanucleus.store.mapped.mapping.MappingHelper.getMappingIndices(MappingHelper.java:35)
        at org.datanucleus.store.mapped.expression.StatementText.applyParametersToStatement(StatementText.java:194)
        at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:233)
        at org.datanucleus.store.rdbms.query.legacy.SQLEvaluator.evaluate(SQLEvaluator.java:115)
        at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.performExecute(JDOQLQuery.java:288)
        at org.datanucleus.store.query.Query.executeQuery(Query.java:1657)
        at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
        at org.datanucleus.store.query.Query.executeWithMap(Query.java:1526)
        at org.datanucleus.jdo.JDOQuery.executeWithMap(JDOQuery.java:334)
        at org.apache.hadoop.hive.metastore.ObjectStore.listMPartitionsByFilter(ObjectStore.java:1329)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilter(ObjectStore.java:1241)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2369)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$40.run(HiveMetaStore.java:2366)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_filter(HiveMetaStore.java:2366)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_filter.process(ThriftHiveMetastore.j
ava:6099)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor.process(ThriftHiveMetastore.java:4789)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$TLoggingProcessor.process(HiveMetaStore.java:3167)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

A null JavaTypeMapping was passed into org.datanucleus.store.mapped.mapping.MappingHelper.(int initialPosition, JavaTypeMapping mapping), that caused NPE.

After digged into the datanucleus source, I found that the null value was born in the constructor of org.datanucleus.store.mapped.expression.SubstringExpression. see

    /**
     * Constructs the substring
     * @param str the String Expression
     * @param begin The start position
     * @param end The end position expression
     **/   
    public SubstringExpression(StringExpression str, NumericExpression begin, NumericExpression end)
    {
        super(str.getQueryExpression());

        st.append("SUBSTRING(").append(str).append(" FROM ")
            .append(begin.add(new IntegerLiteral(qs, mapping, BigInteger.ONE)))
            .append(" FOR ").append(end.sub(begin)).append(')');
    }

The field mapping hasn't been instanced at that moment.

How do you deal with such a external bug?

Issue Links

• blocks: HCATALOG-209 Support for partition-range queries from HCat command-line.
• depends upon: HIVE-2084 Upgrade datanucleus from 2.0.3 to a more recent version (3.?)
• relates to: HIVE-3632 Upgrade datanucleus to support JDK7
• relates to: PIG-2339 HCatLoader loads all the partitions in a partitioned table even though a filter clause on the partitions is specified in the Pig script