Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-2736

Hive UDFs cannot emit binary constants

    Details

      Description

      I recently wrote a UDF which emits BINARY values (as implemented in HIVE-2380). When testing this, I encountered the following exception (because I was evaluating f(g(constant string))) and g() was emitting a BytesWritable type.

      FAILED: Hive Internal Error: java.lang.RuntimeException(Internal error: Cannot find ConstantObjectInspector for BINARY)
      java.lang.RuntimeException: Internal error: Cannot find ConstantObjectInspector for BINARY
      at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.getPrimitiveWritableConstantObjectInspector(PrimitiveObjectInspectorFactory.java:196)
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:899)
      at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:128)
      at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:214)
      at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:684)
      at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:805)
      at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
      at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:161)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:7708)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2301)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2103)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:6126)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:6097)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:6723)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7484)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:889)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

      It looks like a pretty simple fix - add a case for BINARY in PrimitiveObjectInspectorFactory.getPrimitiveWritableConstantObjectInspector() and implement a WritableConstantByteArrayObjectInspector class (almost identical to the others). I'm happy to do this, although this is my first foray into the world of contributing to FOSS so I might end up asking a few stupid questions.

        Attachments

        1. HIVE-2736.1.patch.txt
          4 kB
          Philip Tromans
        2. HIVE-2736.2.patch.txt
          5 kB
          Philip Tromans

          Activity

            People

            • Assignee:
              philip.tromans Philip Tromans
              Reporter:
              philip.tromans Philip Tromans
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 4h
                4h
                Remaining:
                Remaining Estimate - 4h
                4h
                Logged:
                Time Spent - Not Specified
                Not Specified