Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2018

Big-Endian (IBM Power7) Spark Serialization issue

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.3.2, 1.4.0
    • Component/s: Deploy
    • Labels:
      None
    • Environment:

      Description

      We have an application run on Spark on Power7 System .
      But we meet an important issue about serialization.
      The example HdfsWordCount can meet the problem.
      ./bin/run-example org.apache.spark.examples.streaming.HdfsWordCount localdir
      We used Power7 (Big-Endian arch) and Redhat 6.4.
      Big-Endian is the main cause since the example ran successfully in another Power-based Little Endian setup.

      here is the exception stack and log:

      Spark Executor Command: "/opt/ibm/java-ppc64-70//bin/java" "-cp" "/home/test1/spark-1.0.0-bin-hadoop2/lib::/home/test1/src/spark-1.0.0-bin-hadoop2/conf:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/test1/src/hadoop-2.3.0-cdh5.0.0/etc/hadoop/:/home/test1/src/hadoop-2.3.0-cdh5.0.0/etc/hadoop/" "-XX:MaxPermSize=128m" "-Xdebug" "-Xrunjdwp:transport=dt_socket,address=99999,server=y,suspend=n" "-Xms512M" "-Xmx512M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://spark@9.186.105.141:60253/user/CoarseGrainedScheduler" "2" "p7hvs7br16" "4" "akka.tcp://sparkWorker@p7hvs7br16:59240/user/Worker" "app-20140604023054-0000"
      ========================================

      14/06/04 02:31:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      14/06/04 02:31:21 INFO spark.SecurityManager: Changing view acls to: test1,yifeng
      14/06/04 02:31:21 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(test1, yifeng)
      14/06/04 02:31:22 INFO slf4j.Slf4jLogger: Slf4jLogger started
      14/06/04 02:31:22 INFO Remoting: Starting remoting
      14/06/04 02:31:22 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@p7hvs7br16:39658]
      14/06/04 02:31:22 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@p7hvs7br16:39658]
      14/06/04 02:31:22 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@9.186.105.141:60253/user/CoarseGrainedScheduler
      14/06/04 02:31:22 INFO worker.WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@p7hvs7br16:59240/user/Worker
      14/06/04 02:31:23 INFO worker.WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@p7hvs7br16:59240/user/Worker
      14/06/04 02:31:24 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
      14/06/04 02:31:24 INFO spark.SecurityManager: Changing view acls to: test1,yifeng
      14/06/04 02:31:24 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(test1, yifeng)
      14/06/04 02:31:24 INFO slf4j.Slf4jLogger: Slf4jLogger started
      14/06/04 02:31:24 INFO Remoting: Starting remoting
      14/06/04 02:31:24 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@p7hvs7br16:58990]
      14/06/04 02:31:24 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@p7hvs7br16:58990]
      14/06/04 02:31:24 INFO spark.SparkEnv: Connecting to MapOutputTracker: akka.tcp://spark@9.186.105.141:60253/user/MapOutputTracker
      14/06/04 02:31:25 INFO spark.SparkEnv: Connecting to BlockManagerMaster: akka.tcp://spark@9.186.105.141:60253/user/BlockManagerMaster
      14/06/04 02:31:25 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140604023125-3f61
      14/06/04 02:31:25 INFO storage.MemoryStore: MemoryStore started with capacity 307.2 MB.
      14/06/04 02:31:25 INFO network.ConnectionManager: Bound socket to port 39041 with id = ConnectionManagerId(p7hvs7br16,39041)
      14/06/04 02:31:25 INFO storage.BlockManagerMaster: Trying to register BlockManager
      14/06/04 02:31:25 INFO storage.BlockManagerMaster: Registered BlockManager
      14/06/04 02:31:25 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-7bce4e43-2833-4666-93af-bd97c327497b
      14/06/04 02:31:25 INFO spark.HttpServer: Starting HTTP Server
      14/06/04 02:31:25 INFO server.Server: jetty-8.y.z-SNAPSHOT
      14/06/04 02:31:26 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:39958
      14/06/04 02:31:26 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 2
      14/06/04 02:31:26 INFO executor.Executor: Running task ID 2
      14/06/04 02:31:26 ERROR executor.Executor: Exception in task ID 2
      java.io.InvalidClassException: scala.reflect.ClassTag$$anon$1; local class incompatible: stream classdesc serialVersionUID = -8102093212602380348, local class serialVersionUID = -4937928798201944954
      at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:678)
      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1678)
      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1573)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1827)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:607)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)

      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:607)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:607)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:607)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:607)

      at java.lang.reflect.Method.invoke(Method.java:607)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
      at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
      at org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
      at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141)
      at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1893)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1852)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
      at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
      at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:781)
      14/06/04 02:31:26 ERROR executor.CoarseGrainedExecutorBackend: Driver Disassociated [akka.tcp://sparkExecutor@p7hvs7br16:39658] -> [akka.tcp://spark@9.186.105.141:60253] disassociated! Shutting down.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tellison Tim Ellison
                Reporter:
                gaoyanjie55 Yanjie Gao
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: