Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1214

fail to run on SparkRunner with VerifyError

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 0.3.0-incubating
    • 0.5.0
    • runner-spark
    • None

    Description

      I see this exception when running with SparkRunner locally. I fix it in my environment by change jackson-module-scala_2.10 to version 2.7.2.

      <dependency>
      <groupId>com.fasterxml.jackson.module</groupId>
      <artifactId>jackson-module-scala_2.10</artifactId>
      <version>2.7.2</version>
      </dependency>

      Here's the exception stack trace:
      Exception in thread "main" java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClass(Unknown Source)
      at java.security.SecureClassLoader.defineClass(Unknown Source)
      at java.net.URLClassLoader.defineClass(Unknown Source)
      at java.net.URLClassLoader.access$100(Unknown Source)
      at java.net.URLClassLoader$1.run(Unknown Source)
      at java.net.URLClassLoader$1.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(Unknown Source)
      at java.lang.ClassLoader.loadClass(Unknown Source)
      at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
      at java.lang.ClassLoader.loadClass(Unknown Source)
      at com.fasterxml.jackson.module.scala.ser.IteratorSerializerModule$class.$init$(IteratorSerializerModule.scala:70)
      at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:19)
      at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:35)
      at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
      at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:81)
      at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
      at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
      at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:706)
      at org.apache.spark.api.java.JavaRDDLike$class.mapPartitions(JavaRDDLike.scala:161)
      at org.apache.spark.api.java.AbstractJavaRDDLike.mapPartitions(JavaRDDLike.scala:46)
      at org.apache.beam.runners.spark.translation.TransformTranslator$7.evaluate(TransformTranslator.java:262)
      at org.apache.beam.runners.spark.translation.TransformTranslator$7.evaluate(TransformTranslator.java:248)
      at org.apache.beam.runners.spark.SparkRunner$Evaluator.doVisitTransform(SparkRunner.java:267)
      at org.apache.beam.runners.spark.SparkRunner$Evaluator.visitPrimitiveTransform(SparkRunner.java:252)
      at org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:224)
      at org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:219)
      at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:101)
      at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292)
      at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:167)
      at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:75)
      at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)

      Attachments

        Issue Links

          Activity

            People

              ksalant Kobi Salant
              mingmxu Mingmin Xu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: