Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6743

H2O AI Library Causes Errors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Resolved
    • Affects Version/s: 1.14.0, 1.15.0
    • Fix Version/s: None
    • Component/s: Functions - Drill
    • Labels:
      None
    • Environment:

      Mac OSX High Sierra

    • Flags:
      Important

      Description

      I've been working on a UDF to use a POJO generated by H2O Ai.  The basic idea is that a person could train a machine learning model in H2O, generate a POJO and then use Drill and the POJO to make predictions from data in Drill.  

      The idea is similar to this: https://github.com/h2oai/h2o-tutorials/tree/master/tutorials/hive_udf_template/hive_multimojo_udf_template, but for Drill.

      This depends on importing the h2o-genmodel.jar (https://mvnrepository.com/artifact/ai.h2o/h2o-genmodel).  However when you even put that JAR file into Drill's site directory or into the /jars/3rdParty you get the following errors:

      I've been speaking with Paul Rogers about this, and he thinks it has to do with a shaded module, but does anyone have any idea how to fix this?  

       

      2018-09-13 23:38:54,163 ERROR [2464d2b3-7a4a-4449-d08e-61283bce78e0:frag:0:0] record.AbstractUnaryRecordBatch: Failure during query
      org.apache.drill.exec.exception.SchemaChangeException: Failure while attempting to load generated class
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:572)}}
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:582)}}
      {{ at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:101)}}
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)}}
      {{ at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)}}
      {{ at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)}}
      {{ at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)}}
      {{ at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:293)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:280)}}
      {{ at java.security.AccessController.doPrivileged(Native Method)}}
      {{ at javax.security.auth.Subject.doAs(Subject.java:422)}}
      {{ at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:280)}}
      {{ at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}}
      {{ at java.lang.Thread.run(Thread.java:745)}}
      Caused by: org.apache.drill.exec.exception.ClassTransformationException: org.apache.drill.shaded.guava.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException
      {{ at org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:197)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.createInstance(CodeCompiler.java:163)}}
      {{ at org.apache.drill.exec.ops.BaseFragmentContext.getImplementationClass(BaseFragmentContext.java:56)}}
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:569)}}
      {{ ... 17 more}}
      Caused by: org.apache.drill.shaded.guava.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache.get(LocalCache.java:4147)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4151)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5140)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:186)}}
      {{ ... 20 more}}
      Caused by: java.lang.IllegalArgumentException
      {{ at org.objectweb.asm.ClassReader.<init>(Unknown Source)}}
      {{ at org.objectweb.asm.ClassReader.<init>(Unknown Source)}}
      {{ at org.apache.drill.exec.compile.AsmUtil.classFromBytes(AsmUtil.java:93)}}
      {{ at org.apache.drill.exec.compile.AsmUtil.isClassBytesOk(AsmUtil.java:80)}}
      {{ at org.apache.drill.exec.compile.MergeAdapter.getMergedClass(MergeAdapter.java:206)}}
      {{ at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:289)}}
      {{ at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:228)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler$CodeGenCompiler.compile(CodeCompiler.java:79)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.makeClass(CodeCompiler.java:229)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.access$300(CodeCompiler.java:41)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:212)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:209)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212)}}
      {{ ... 24 more}}
      2018-09-13 23:38:54,170 ERROR [2464d2b3-7a4a-4449-d08e-61283bce78e0:frag:0:0] fragment.FragmentExecutor: SYSTEM ERROR: IllegalArgumentException

      Fragment 0:0

      [Error Id: 90ee4178-17b7-4566-af59-85b99fb64d22 on charles-mbp.fios-router.home:31010]
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalArgumentException

      Fragment 0:0

      [Error Id: 90ee4178-17b7-4566-af59-85b99fb64d22 on charles-mbp.fios-router.home:31010]
      {{ at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:360)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:215)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:326)}}
      {{ at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}}
      {{ at java.lang.Thread.run(Thread.java:745)}}
      Caused by: org.apache.drill.exec.exception.SchemaChangeException: Failure while attempting to load generated class
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:572)}}
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:582)}}
      {{ at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:101)}}
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)}}
      {{ at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)}}
      {{ at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)}}
      {{ at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)}}
      {{ at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:293)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:280)}}
      {{ at java.security.AccessController.doPrivileged(Native Method)}}
      {{ at javax.security.auth.Subject.doAs(Subject.java:422)}}
      {{ at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)}}
      {{ at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:280)}}
      {{ ... 4 more}}
      Caused by: org.apache.drill.exec.exception.ClassTransformationException: org.apache.drill.shaded.guava.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException
      {{ at org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:197)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.createInstance(CodeCompiler.java:163)}}
      {{ at org.apache.drill.exec.ops.BaseFragmentContext.getImplementationClass(BaseFragmentContext.java:56)}}
      {{ at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:569)}}
      {{ ... 17 more}}
      Caused by: org.apache.drill.shaded.guava.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache.get(LocalCache.java:4147)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4151)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5140)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:186)}}
      {{ ... 20 more}}
      Caused by: java.lang.IllegalArgumentException
      {{ at org.objectweb.asm.ClassReader.<init>(Unknown Source)}}
      {{ at org.objectweb.asm.ClassReader.<init>(Unknown Source)}}
      {{ at org.apache.drill.exec.compile.AsmUtil.classFromBytes(AsmUtil.java:93)}}
      {{ at org.apache.drill.exec.compile.AsmUtil.isClassBytesOk(AsmUtil.java:80)}}
      {{ at org.apache.drill.exec.compile.MergeAdapter.getMergedClass(MergeAdapter.java:206)}}
      {{ at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:289)}}
      {{ at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:228)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler$CodeGenCompiler.compile(CodeCompiler.java:79)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.makeClass(CodeCompiler.java:229)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler.access$300(CodeCompiler.java:41)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:212)}}
      {{ at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:209)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299)}}
      {{ at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212)}}
      {{ ... 24 more}}
      Error: SYSTEM ERROR: IllegalArgumentException

      Fragment 0:0

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cgivre Charles Givre
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: