Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24086

Exception while executing spark streaming examples

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: Examples
    • Labels:
      None

      Description

      After running mvn clean package, I tried to execute one of the spark example program JavaDirectKafkaWordCount.java but throws following exeception.

      [cloud-user@server-2 examples]$ run-example streaming.JavaDirectKafkaWordCount 192.168.0.4:9092 msu
      2018-04-25 09:39:22 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2018-04-25 09:39:22 INFO SparkContext:54 - Running Spark version 2.3.0
      2018-04-25 09:39:22 INFO SparkContext:54 - Submitted application: JavaDirectKafkaWordCount
      2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls to: cloud-user
      2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls to: cloud-user
      2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls groups to:
      2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls groups to:
      2018-04-25 09:39:22 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloud-user); groups with view permissions: Set(); users with modify permissions: Set(cloud-user); groups with modify permissions: Set()
      2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'sparkDriver' on port 59333.
      2018-04-25 09:39:23 INFO SparkEnv:54 - Registering MapOutputTracker
      2018-04-25 09:39:23 INFO SparkEnv:54 - Registering BlockManagerMaster
      2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
      2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
      2018-04-25 09:39:23 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-6fc11fc1-f638-42ea-a9df-dc01fb81b7b6
      2018-04-25 09:39:23 INFO MemoryStore:54 - MemoryStore started with capacity 366.3 MB
      2018-04-25 09:39:23 INFO SparkEnv:54 - Registering OutputCommitCoordinator
      2018-04-25 09:39:23 INFO log:192 - Logging initialized @1825ms
      2018-04-25 09:39:23 INFO Server:346 - jetty-9.3.z-SNAPSHOT
      2018-04-25 09:39:23 INFO Server:414 - Started @1900ms
      2018-04-25 09:39:23 INFO AbstractConnector:278 - Started ServerConnector@6813a331{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
      2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4f7c0be3{/jobs,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4cfbaf4{/jobs/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58faa93b{/jobs/job,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@127d7908{/jobs/job/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b9c69a9{/stages,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6622a690{/stages/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30b9eadd{/stages/stage,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3249a1ce{/stages/stage/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4dd94a58{/stages/pool,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2f4919b0{/stages/pool/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@a8a8b75{/storage,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@75b21c3b{/storage/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@72be135f{/storage/rdd,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@155d1021{/storage/rdd/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4bd2f0dc{/environment,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2e647e59{/environment/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c42b421{/executors,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51e37590{/executors/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@deb3b60{/executors/threadDump,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@701a32{/executors/threadDump/json,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39aa45a1{/static,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@294bdeb4{/,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5300f14a{/api,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54acff7d{/jobs/job/kill,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7bc9e6ab{/stages/stage/kill,null,AVAILABLE,@Spark}
      2018-04-25 09:39:23 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://server-2:4040
      2018-04-25 09:39:23 INFO SparkContext:54 - Added JAR file:///home/cloud-user/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar at spark://server-2:59333/jars/spark-examples_2.11-2.3.0.jar with timestamp 1524663563504
      2018-04-25 09:39:23 INFO SparkContext:54 - Added JAR file:///home/cloud-user/spark-2.3.0-bin-hadoop2.7/examples/jars/scopt_2.11-3.7.0.jar at spark://server-2:59333/jars/scopt_2.11-3.7.0.jar with timestamp 1524663563505
      2018-04-25 09:39:23 INFO Executor:54 - Starting executor ID driver on host localhost
      2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56363.
      2018-04-25 09:39:23 INFO NettyBlockTransferService:54 - Server created on server-2:56363
      2018-04-25 09:39:23 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
      2018-04-25 09:39:23 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, server-2, 56363, None)
      2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - Registering block manager server-2:56363 with 366.3 MB RAM, BlockManagerId(driver, server-2, 56363, None)
      2018-04-25 09:39:23 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, server-2, 56363, None)
      2018-04-25 09:39:23 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, server-2, 56363, None)
      2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e2916c3{/metrics/json,null,AVAILABLE,@Spark}
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka010/LocationStrategies
      at org.apache.spark.examples.streaming.JavaDirectKafkaWordCount.main(JavaDirectKafkaWordCount.java:76)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka010.LocationStrategies
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 11 more
      2018-04-25 09:39:23 INFO SparkContext:54 - Invoking stop() from shutdown hook
      2018-04-25 09:39:23 INFO AbstractConnector:318 - Stopped Spark@6813a331{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
      2018-04-25 09:39:23 INFO SparkUI:54 - Stopped Spark web UI at http://server-2:4040
      2018-04-25 09:39:23 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
      2018-04-25 09:39:23 INFO MemoryStore:54 - MemoryStore cleared
      2018-04-25 09:39:23 INFO BlockManager:54 - BlockManager stopped
      2018-04-25 09:39:23 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
      2018-04-25 09:39:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
      2018-04-25 09:39:23 INFO SparkContext:54 - Successfully stopped SparkContext
      2018-04-25 09:39:23 INFO ShutdownHookManager:54 - Shutdown hook called
      2018-04-25 09:39:23 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-edc94694-ab74-4b66-9ef2-10d28b3f5359
      2018-04-25 09:39:23 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-d57ca1de-b096-4036-ad4b-ed97295443c4
      
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hasan4791 Chandra Hasan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: