Kafka
  1. Kafka
  2. KAFKA-797

Cannot start Kafka 0.8 per Quick Start instructions

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Checked out latest 0.8 (b5edbb193b33ebf0b3056935a537967ff21478a6), and ran:

      ./sbt update
      ./sbt package

      Both complete successfully. Then I try to start the server:

      $ ./bin/kafka-server-start.sh config/server.properties
      Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
      at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
      at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
      at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
      at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
      at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
      at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      at kafka.Kafka.main(Kafka.scala)
      Caused by: java.lang.ClassNotFoundException: scala.ScalaObject
      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      ... 25 more

        Activity

        Hide
        Bob added a comment -

        While I also cannot start 0.8 per quick start instructions, my problem is slightly different. I can start zookeeper but get this error trying to start any broker
        bin/kafka-server-start.sh config/server1.properties

        [2013-03-08 16:55:19,287] ERROR Error while electing or becoming leader on broker 1 (kafka.server.ZookeeperLeaderElector)
        kafka.common.KafkaException: Can't parse json string: null
        at kafka.utils.Json$.liftedTree1$1(Json.scala:20)
        at kafka.utils.Json$.parseFull(Json.scala:16)
        at kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:484)
        at kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:480)
        at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
        at scala.collection.immutable.List.foreach(List.scala:45)
        at kafka.utils.ZkUtils$.getReplicaAssignmentForTopics(ZkUtils.scala:480)
        at kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:451)
        at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:225)
        at kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:87)
        at kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:53)
        at kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:106)
        at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:549)
        at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
        Caused by: java.lang.NullPointerException
        at scala.util.parsing.combinator.lexical.Scanners$Scanner.<init>(Scanners.scala:52)
        at scala.util.parsing.json.JSON$.parseRaw(JSON.scala:71)
        at scala.util.parsing.json.JSON$.parseFull(JSON.scala:85)
        at kafka.utils.Json$.liftedTree1$1(Json.scala:17)
        ... 13 more

        Show
        Bob added a comment - While I also cannot start 0.8 per quick start instructions, my problem is slightly different. I can start zookeeper but get this error trying to start any broker bin/kafka-server-start.sh config/server1.properties [2013-03-08 16:55:19,287] ERROR Error while electing or becoming leader on broker 1 (kafka.server.ZookeeperLeaderElector) kafka.common.KafkaException: Can't parse json string: null at kafka.utils.Json$.liftedTree1$1(Json.scala:20) at kafka.utils.Json$.parseFull(Json.scala:16) at kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:484) at kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:480) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at kafka.utils.ZkUtils$.getReplicaAssignmentForTopics(ZkUtils.scala:480) at kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:451) at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:225) at kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:87) at kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:53) at kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:106) at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:549) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) Caused by: java.lang.NullPointerException at scala.util.parsing.combinator.lexical.Scanners$Scanner.<init>(Scanners.scala:52) at scala.util.parsing.json.JSON$.parseRaw(JSON.scala:71) at scala.util.parsing.json.JSON$.parseFull(JSON.scala:85) at kafka.utils.Json$.liftedTree1$1(Json.scala:17) ... 13 more
        Hide
        Swapnil Ghike added a comment -

        Bob, are you using 0.8 HEAD? We did a bunch of changes related to parsing the zk data. You can see the changes at KAFKA-755, KAFKA-779 and KAFKA-776.

        If your zookeeper has data according to the old format, then you may need to clean up all the zookeeper data and restart zookeeper and kafka server.

        Show
        Swapnil Ghike added a comment - Bob, are you using 0.8 HEAD? We did a bunch of changes related to parsing the zk data. You can see the changes at KAFKA-755 , KAFKA-779 and KAFKA-776 . If your zookeeper has data according to the old format, then you may need to clean up all the zookeeper data and restart zookeeper and kafka server.
        Hide
        Bob added a comment -

        Swapni,

        Per instructions, I cloned it as such
        git clone https://git-wip-us.apache.org/repos/asf/kafka.git

        Is that not off head? I cloned this yesterday, so I shouldn't have data in the old format, incidentally, how do I check if I have zookeeper data? Sorry for the question, I am new to Zookeeper and Kafka though I know Scala and is trying to understand / debug ZookeeperLeaderElector.

        Bob

        Show
        Bob added a comment - Swapni, Per instructions, I cloned it as such git clone https://git-wip-us.apache.org/repos/asf/kafka.git Is that not off head? I cloned this yesterday, so I shouldn't have data in the old format, incidentally, how do I check if I have zookeeper data? Sorry for the question, I am new to Zookeeper and Kafka though I know Scala and is trying to understand / debug ZookeeperLeaderElector. Bob
        Hide
        Bob added a comment -

        Swapni,

        Caveat, I am still new to the code but tracing through the code path, I found that the exception is caused while trying to create an ephemeral node but something isn't right because the ephemeral node is created in ZkUtils.createEphemeralPathExpectConflict method, client.toString() shows the node created as org.I0Itec.zkclient.ZkClient@e0d5eb7. However in the calling ZookeeperLeaderElector.elect method, it threw e2 exception after calling createEphemeralPathExpectConflict which passed!

        Maybe you can shed some light on why this is happening.

        Thanks,
        Bob

        Show
        Bob added a comment - Swapni, Caveat, I am still new to the code but tracing through the code path, I found that the exception is caused while trying to create an ephemeral node but something isn't right because the ephemeral node is created in ZkUtils.createEphemeralPathExpectConflict method, client.toString() shows the node created as org.I0Itec.zkclient.ZkClient@e0d5eb7. However in the calling ZookeeperLeaderElector.elect method, it threw e2 exception after calling createEphemeralPathExpectConflict which passed! Maybe you can shed some light on why this is happening. Thanks, Bob
        Hide
        Bob added a comment -

        Ok, I found 2 bugs in ZkUtils.getReplicaAssignmentForTopics and getPartitionAssignmentForTopics. It has to do with improperly handled Java null string, here's my workaround, now the quickstart example works end to end. BTW is it possible for me to fork this repo and submit a pull request?

        def getReplicaAssignmentForTopics(zkClient: ZkClient, topics: Seq[String]): mutable.Map[TopicAndPartition, Seq[Int]] = {
        val ret = new mutable.HashMap[TopicAndPartition, Seq[Int]]
        topics.foreach { topic =>
        val jsonPartitionMapOpt = readDataMaybeNull(zkClient, getTopicPath(topic))._1
        jsonPartitionMapOpt match {
        case Some(jsonPartitionMap) =>
        //<<<<<<null check>>>>>>>>
        if (jsonPartitionMap != null) {
        Json.parseFull(jsonPartitionMap) match {
        case Some(m) => m.asInstanceOf[Map[String, Any]].get("partitions") match {
        case Some(repl) =>
        val replicaMap = repl.asInstanceOf[Map[String, Seq[Int]]]
        for ((partition, replicas) <- replicaMap)

        { ret.put(TopicAndPartition(topic, partition.toInt), replicas) debug("Replicas assigned to topic [%s], partition [%s] are [%s]".format(topic, partition, replicas)) }

        case None =>
        }
        case None =>
        }
        }
        case None =>
        }
        }
        ret
        }

        def getPartitionAssignmentForTopics(zkClient: ZkClient, topics: Seq[String]): mutable.Map[String, collection.Map[Int, Seq[Int]]] = {
        val ret = new mutable.HashMap[String, Map[Int, Seq[Int]]]()
        topics.foreach { topic =>
        val jsonPartitionMapOpt = readDataMaybeNull(zkClient, getTopicPath(topic))._1
        val partitionMap = jsonPartitionMapOpt match {
        case Some(jsonPartitionMap) =>
        //<<<<<<null check>>>>>>>>
        if (jsonPartitionMap != null) {
        Json.parseFull(jsonPartitionMap) match {
        case Some(m) => m.asInstanceOf[Map[String, Any]].get("partitions") match

        { case Some(replicaMap) => val m1 = replicaMap.asInstanceOf[Map[String, Seq[Int]]] m1.map(p => (p._1.toInt, p._2)) case None => Map[Int, Seq[Int]]() }

        case None => Map[Int, Seq[Int]]()
        }
        } else Map[Int, Seq[Int]]() //<<<<<set default
        case None => Map[Int, Seq[Int]]()
        }
        debug("Partition map for /brokers/topics/%s is %s".format(topic, partitionMap))
        ret += (topic -> partitionMap)
        }
        ret
        }

        Show
        Bob added a comment - Ok, I found 2 bugs in ZkUtils.getReplicaAssignmentForTopics and getPartitionAssignmentForTopics. It has to do with improperly handled Java null string, here's my workaround, now the quickstart example works end to end. BTW is it possible for me to fork this repo and submit a pull request? def getReplicaAssignmentForTopics(zkClient: ZkClient, topics: Seq [String] ): mutable.Map[TopicAndPartition, Seq [Int] ] = { val ret = new mutable.HashMap[TopicAndPartition, Seq [Int] ] topics.foreach { topic => val jsonPartitionMapOpt = readDataMaybeNull(zkClient, getTopicPath(topic))._1 jsonPartitionMapOpt match { case Some(jsonPartitionMap) => //<<<<<<null check>>>>>>>> if (jsonPartitionMap != null) { Json.parseFull(jsonPartitionMap) match { case Some(m) => m.asInstanceOf[Map [String, Any] ].get("partitions") match { case Some(repl) => val replicaMap = repl.asInstanceOf[Map[String, Seq [Int] ]] for ((partition, replicas) <- replicaMap) { ret.put(TopicAndPartition(topic, partition.toInt), replicas) debug("Replicas assigned to topic [%s], partition [%s] are [%s]".format(topic, partition, replicas)) } case None => } case None => } } case None => } } ret } def getPartitionAssignmentForTopics(zkClient: ZkClient, topics: Seq [String] ): mutable.Map[String, collection.Map[Int, Seq [Int] ]] = { val ret = new mutable.HashMap[String, Map[Int, Seq [Int] ]]() topics.foreach { topic => val jsonPartitionMapOpt = readDataMaybeNull(zkClient, getTopicPath(topic))._1 val partitionMap = jsonPartitionMapOpt match { case Some(jsonPartitionMap) => //<<<<<<null check>>>>>>>> if (jsonPartitionMap != null) { Json.parseFull(jsonPartitionMap) match { case Some(m) => m.asInstanceOf[Map [String, Any] ].get("partitions") match { case Some(replicaMap) => val m1 = replicaMap.asInstanceOf[Map[String, Seq[Int]]] m1.map(p => (p._1.toInt, p._2)) case None => Map[Int, Seq[Int]]() } case None => Map[Int, Seq [Int] ]() } } else Map[Int, Seq [Int] ]() //<<<<<set default case None => Map[Int, Seq [Int] ]() } debug("Partition map for /brokers/topics/%s is %s".format(topic, partitionMap)) ret += (topic -> partitionMap) } ret }
        Hide
        Matan Safriel added a comment -

        Back to the original topic, David, the problem you encountered (Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject) may seem to be related to Java not being able to run Scala on your machine.
        Can you share the versions of SBT and Java (java -version)?

        Also you may want to try in ./sbt:
        clean
        ++ 2.9.2
        package

        HTH

        Show
        Matan Safriel added a comment - Back to the original topic, David, the problem you encountered (Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject) may seem to be related to Java not being able to run Scala on your machine. Can you share the versions of SBT and Java (java -version)? Also you may want to try in ./sbt: clean ++ 2.9.2 package HTH
        Hide
        David Arthur added a comment - - edited

        Matan, after running these commands, and trying the kafka-server-start.sh script, i get

        Exception in thread "main" java.lang.NoClassDefFoundError: kafka/Kafka
        Caused by: java.lang.ClassNotFoundException: kafka.Kafka
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

        Also,

        java version "1.6.0_41"
        Java(TM) SE Runtime Environment (build 1.6.0_41-b02-445-11M4107)
        Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01-445, mixed mode)

        Show
        David Arthur added a comment - - edited Matan, after running these commands, and trying the kafka-server-start.sh script, i get Exception in thread "main" java.lang.NoClassDefFoundError: kafka/Kafka Caused by: java.lang.ClassNotFoundException: kafka.Kafka at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Also, java version "1.6.0_41" Java(TM) SE Runtime Environment (build 1.6.0_41-b02-445-11M4107) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01-445, mixed mode)
        Hide
        Swapnil Ghike added a comment -

        Bob, yes, if you cloned the repo two days ago, then you should have trunk HEAD. The 0.8 branch contains one more Zk related change and some other additional bug fixes. Trunk is where we are developing for post 0.8 changes, and we periodically merge any changes in 0.8 to trunk.

        The case for None should take care of null because null converted to an Option is None. If the problem persists on the 0.8 branch without your local fix, can you please file a separate JIRA for it? It may be a bug. Thanks.

        Show
        Swapnil Ghike added a comment - Bob, yes, if you cloned the repo two days ago, then you should have trunk HEAD. The 0.8 branch contains one more Zk related change and some other additional bug fixes. Trunk is where we are developing for post 0.8 changes, and we periodically merge any changes in 0.8 to trunk. The case for None should take care of null because null converted to an Option is None. If the problem persists on the 0.8 branch without your local fix, can you please file a separate JIRA for it? It may be a bug. Thanks.
        Hide
        Bob added a comment -

        Hi Sawpnil,

        null isn't handled as expected by None, I have submitted a new JIRA and a patch at
        https://issues.apache.org/jira/browse/KAFKA-799

        Show
        Bob added a comment - Hi Sawpnil, null isn't handled as expected by None, I have submitted a new JIRA and a patch at https://issues.apache.org/jira/browse/KAFKA-799
        Hide
        Jun Rao added a comment -

        David,

        Did you do a fresh checkout of 0.8?

        Show
        Jun Rao added a comment - David, Did you do a fresh checkout of 0.8?
        Hide
        David Arthur added a comment -

        Jun, here I have done a fresh clone:

        mumrah@davids-mbp /tmp:$ git clone git://github.com/apache/kafka.git
        Cloning into kafka...
        remote: Counting objects: 11895, done.
        remote: Compressing objects: 100% (3709/3709), done.
        remote: Total 11895 (delta 7192), reused 11281 (delta 6782)
        Receiving objects: 100% (11895/11895), 13.91 MiB | 963 KiB/s, done.
        Resolving deltas: 100% (7192/7192), done.
        mumrah@davids-mbp /tmp:$ cd kafka/
        mumrah@davids-mbp /tmp/kafka: (trunk)$ git checkout -b 0.8 origin/0.8
        Branch 0.8 set up to track remote branch 0.8 from origin by rebasing.
        Switched to a new branch '0.8'
        mumrah@davids-mbp /tmp/kafka: (0.8)$ ./sbt

        // Bunch of Ivy stuff

        > update

        // More Ivy stuff

        > package

        > mumrah@davids-mbp /tmp/kafka: (0.8)$ ./bin/kafka-server-start.sh config/server.properties
        Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at kafka.Kafka.main(Kafka.scala)
        Caused by: java.lang.ClassNotFoundException: scala.ScalaObject
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 25 more

        mumrah@davids-mbp /tmp/kafka: (0.8)$ git log HEAD~1..HEAD
        commit da1dc17887e536824df35958c6abdc14828fb4cb
        Author: Neha Narkhede <neha.narkhede@gmail.com>
        Date: Sat Mar 9 14:53:22 2013 -0800

        KAFKA-798 Use biased histograms instead of uniform histograms in KafkaMetricsGroup; reviewed by Neha Narkhede

        Show
        David Arthur added a comment - Jun, here I have done a fresh clone: mumrah@davids-mbp /tmp:$ git clone git://github.com/apache/kafka.git Cloning into kafka... remote: Counting objects: 11895, done. remote: Compressing objects: 100% (3709/3709), done. remote: Total 11895 (delta 7192), reused 11281 (delta 6782) Receiving objects: 100% (11895/11895), 13.91 MiB | 963 KiB/s, done. Resolving deltas: 100% (7192/7192), done. mumrah@davids-mbp /tmp:$ cd kafka/ mumrah@davids-mbp /tmp/kafka: (trunk)$ git checkout -b 0.8 origin/0.8 Branch 0.8 set up to track remote branch 0.8 from origin by rebasing. Switched to a new branch '0.8' mumrah@davids-mbp /tmp/kafka: (0.8)$ ./sbt // Bunch of Ivy stuff > update // More Ivy stuff > package > mumrah@davids-mbp /tmp/kafka: (0.8)$ ./bin/kafka-server-start.sh config/server.properties Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at kafka.Kafka.main(Kafka.scala) Caused by: java.lang.ClassNotFoundException: scala.ScalaObject at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 25 more mumrah@davids-mbp /tmp/kafka: (0.8)$ git log HEAD~1..HEAD commit da1dc17887e536824df35958c6abdc14828fb4cb Author: Neha Narkhede <neha.narkhede@gmail.com> Date: Sat Mar 9 14:53:22 2013 -0800 KAFKA-798 Use biased histograms instead of uniform histograms in KafkaMetricsGroup; reviewed by Neha Narkhede
        Hide
        David Arthur added a comment -

        So upon further inspection, I see a couple of problems:

        1. 2.8.0 is hard coded in the kafka-run-class.sh script
        2. Even when compiling with 2.8.0, the script doesn't find scala-compiler or scala-library in the Ivy cache. It builds: /Users/mumrah/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.8.0.jar and ~/Users/mumrah/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.8.0.jar, but neither exist

        Building up classpaths from the Ivy cache without really using Ivy is very brittle, IMO.

        I would be willing to work on a fix for this, but I'd need clarification on the placement and usage of the libraries throughout the project. E.g., what's the difference between core/lib, lib, core/target, target?

        Show
        David Arthur added a comment - So upon further inspection, I see a couple of problems: 1. 2.8.0 is hard coded in the kafka-run-class.sh script 2. Even when compiling with 2.8.0, the script doesn't find scala-compiler or scala-library in the Ivy cache. It builds: /Users/mumrah/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.8.0.jar and ~/Users/mumrah/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.8.0.jar, but neither exist Building up classpaths from the Ivy cache without really using Ivy is very brittle, IMO. I would be willing to work on a fix for this, but I'd need clarification on the placement and usage of the libraries throughout the project. E.g., what's the difference between core/lib, lib, core/target, target?
        Hide
        Neha Narkhede added a comment -

        David,

        When you start sbt, it automatically downloads and places the scala jars under project/boot/<scala-version>/lib. Are you saying that somehow that didn't happen for you ?

        Show
        Neha Narkhede added a comment - David, When you start sbt, it automatically downloads and places the scala jars under project/boot/<scala-version>/lib. Are you saying that somehow that didn't happen for you ?
        Hide
        David Arthur added a comment -

        Neha,

        That's correct. My project dir has:

        Build.scala build.properties project
        build plugins.sbt target

        but no boot

        Show
        David Arthur added a comment - Neha, That's correct. My project dir has: Build.scala build.properties project build plugins.sbt target but no boot
        Hide
        David Arthur added a comment -

        I have to run "./sbt update" in order for scala-libarary and scala-compiler to appear in ./project/boot. I was able to run Kafka with the following command:

        java -verbose:class -Dlog4j.debug=true -cp .:./project/boot/scala-2.8.0/lib/:./core/lib_managed/scala_2.8.0/compile/:./core/lib/*:./core/target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar:config/log4j.properties:/Users/mumrah/.ivy2/cache/org.apache.zookeeper/zookeeper/jars/zookeeper-3.3.4.jar:/Users/mumrah/.ivy2/cache/net.sf.jopt-simple/jopt-simple/jars/jopt-simple-3.2.jar:/Users/mumrah/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.6.4.jar:/Users/mumrah/.ivy2/cache/log4j/log4j/jars/log4j-1.2.15.jar kafka.Kafka config/server.properties

        Here are the jars in my project directory

        $ find . | grep 'jar$'
        ... excluding ./contrib
        ./core/lib/metrics-annotation-3.0.0-c0c8be71.jar
        ./core/lib/metrics-core-3.0.0-c0c8be71.jar
        ./core/lib/zkclient-20120522.jar
        ./core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar
        ./core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar
        ./core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar
        ./core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar
        ./core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar
        ./core/lib_managed/scala_2.8.0/test/cglib-nodep-2.2.jar
        ./core/lib_managed/scala_2.8.0/test/easymock-3.0.jar
        ./core/lib_managed/scala_2.8.0/test/junit-4.1.jar
        ./core/lib_managed/scala_2.8.0/test/objenesis-1.2.jar
        ./core/lib_managed/scala_2.8.0/test/scalatest-1.2.jar
        ./core/target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar
        ./core/target/scala-2.9.2/kafka_2.9.2-0.8-SNAPSHOT.jar
        ./examples/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar
        ./examples/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar
        ./examples/target/kafka-java-examples-0.8-SNAPSHOT.jar
        ./lib/apache-rat-0.8.jar
        ./lib/sbt-launch.jar
        ./perf/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar
        ./perf/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar
        ./perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8-SNAPSHOT.jar
        ./perf/target/scala-2.9.2/kafka-perf_2.9.2-0.8-SNAPSHOT.jar
        ./project/boot/scala-2.7.7/lib/scala-compiler.jar
        ./project/boot/scala-2.7.7/lib/scala-library.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/classpath_2.7.7-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compile_2.7.7-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-bin_2.7.7.final/compiler-interface-bin-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-bin_2.8.0.final/compiler-interface-bin-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-bin_2.8.1.final/compiler-interface-bin-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-src/compiler-interface-src-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-src/jline-0.9.94.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/control_2.7.7-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/io_2.7.7-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/ivy-2.2.0.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/ivy_2.7.7-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/jsch-0.1.31.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/launcher-interface-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/sbt_2.7.7-0.7.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/test-interface-0.5.jar
        ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/xsbti/interface-0.7.5.jar
        ./project/boot/scala-2.8.0/lib/scala-compiler.jar
        ./project/boot/scala-2.8.0/lib/scala-library.jar
        ./project/plugins/lib_managed/scala_2.7.7/sbt-idea-core_2.7.7-0.1-SNAPSHOT.jar
        ./project/plugins/lib_managed/scala_2.7.7/sbt-idea-plugin-0.1-SNAPSHOT.jar
        ./system_test/migration_tool_testsuite/0.7/lib/kafka-0.7.0.jar
        ./system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar
        ./system_test/migration_tool_testsuite/0.7/lib/zkclient-0.1.jar
        ./target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar
        ./target/scala-2.9.2/kafka_2.9.2-0.8-SNAPSHOT.jar

        Show
        David Arthur added a comment - I have to run "./sbt update" in order for scala-libarary and scala-compiler to appear in ./project/boot. I was able to run Kafka with the following command: java -verbose:class -Dlog4j.debug=true -cp .:./project/boot/scala-2.8.0/lib/ :./core/lib_managed/scala_2.8.0/compile/ :./core/lib/*:./core/target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar:config/log4j.properties:/Users/mumrah/.ivy2/cache/org.apache.zookeeper/zookeeper/jars/zookeeper-3.3.4.jar:/Users/mumrah/.ivy2/cache/net.sf.jopt-simple/jopt-simple/jars/jopt-simple-3.2.jar:/Users/mumrah/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.6.4.jar:/Users/mumrah/.ivy2/cache/log4j/log4j/jars/log4j-1.2.15.jar kafka.Kafka config/server.properties Here are the jars in my project directory $ find . | grep 'jar$' ... excluding ./contrib ./core/lib/metrics-annotation-3.0.0-c0c8be71.jar ./core/lib/metrics-core-3.0.0-c0c8be71.jar ./core/lib/zkclient-20120522.jar ./core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar ./core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar ./core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar ./core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar ./core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar ./core/lib_managed/scala_2.8.0/test/cglib-nodep-2.2.jar ./core/lib_managed/scala_2.8.0/test/easymock-3.0.jar ./core/lib_managed/scala_2.8.0/test/junit-4.1.jar ./core/lib_managed/scala_2.8.0/test/objenesis-1.2.jar ./core/lib_managed/scala_2.8.0/test/scalatest-1.2.jar ./core/target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar ./core/target/scala-2.9.2/kafka_2.9.2-0.8-SNAPSHOT.jar ./examples/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar ./examples/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar ./examples/target/kafka-java-examples-0.8-SNAPSHOT.jar ./lib/apache-rat-0.8.jar ./lib/sbt-launch.jar ./perf/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar ./perf/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar ./perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8-SNAPSHOT.jar ./perf/target/scala-2.9.2/kafka-perf_2.9.2-0.8-SNAPSHOT.jar ./project/boot/scala-2.7.7/lib/scala-compiler.jar ./project/boot/scala-2.7.7/lib/scala-library.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/classpath_2.7.7-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compile_2.7.7-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-bin_2.7.7.final/compiler-interface-bin-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-bin_2.8.0.final/compiler-interface-bin-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-bin_2.8.1.final/compiler-interface-bin-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-src/compiler-interface-src-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/compiler-interface-src/jline-0.9.94.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/control_2.7.7-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/io_2.7.7-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/ivy-2.2.0.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/ivy_2.7.7-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/jsch-0.1.31.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/launcher-interface-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/sbt_2.7.7-0.7.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/test-interface-0.5.jar ./project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.5/xsbti/interface-0.7.5.jar ./project/boot/scala-2.8.0/lib/scala-compiler.jar ./project/boot/scala-2.8.0/lib/scala-library.jar ./project/plugins/lib_managed/scala_2.7.7/sbt-idea-core_2.7.7-0.1-SNAPSHOT.jar ./project/plugins/lib_managed/scala_2.7.7/sbt-idea-plugin-0.1-SNAPSHOT.jar ./system_test/migration_tool_testsuite/0.7/lib/kafka-0.7.0.jar ./system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar ./system_test/migration_tool_testsuite/0.7/lib/zkclient-0.1.jar ./target/scala-2.8.0/kafka_2.8.0-0.8-SNAPSHOT.jar ./target/scala-2.9.2/kafka_2.9.2-0.8-SNAPSHOT.jar
        Hide
        Phil Hargett added a comment -

        Same issue here. I cloned the repo and did a checkout of the 0.8 branch (yesterday) before starting a build using update / package. Both the scala-library and scala-compiler jars were missing; had to supply them myself to run.

        Also: FYI, Bob, the null pointer exception parsing JSON looks like the issue I hit related to bug 686.

        Show
        Phil Hargett added a comment - Same issue here. I cloned the repo and did a checkout of the 0.8 branch (yesterday) before starting a build using update / package. Both the scala-library and scala-compiler jars were missing; had to supply them myself to run. Also: FYI, Bob, the null pointer exception parsing JSON looks like the issue I hit related to bug 686.
        Hide
        Dragos Manolescu added a comment -

        I second David's and Phil's observations, the scala-compiler and scala-library JARs are not downloaded by sbt: I wiped the ivy cache and ran into the same problem. I also agree that referencing ~/.ivy2/cache from the classpath is asking for trouble

        Poking around the build file I noticed that the sbt-assembly plugin is already there, though nothing seems to be using it. Perhaps leveraging it to package all dependencies into a single jar would be more robust than extending the classpath with individual JARs in kafka-run-class.sh?

        Show
        Dragos Manolescu added a comment - I second David's and Phil's observations, the scala-compiler and scala-library JARs are not downloaded by sbt: I wiped the ivy cache and ran into the same problem. I also agree that referencing ~/.ivy2/cache from the classpath is asking for trouble Poking around the build file I noticed that the sbt-assembly plugin is already there, though nothing seems to be using it. Perhaps leveraging it to package all dependencies into a single jar would be more robust than extending the classpath with individual JARs in kafka-run-class.sh?
        Hide
        Bob added a comment -

        Phil, yep, it's the same issue. I filed a separate bug on it
        https://issues.apache.org/jira/browse/KAFKA-797

        Show
        Bob added a comment - Phil, yep, it's the same issue. I filed a separate bug on it https://issues.apache.org/jira/browse/KAFKA-797
        Hide
        Dave DeMaagd added a comment -

        KAFKA-733 is already out there for the assembly stuff, though there was some question about the conflict handling.

        Show
        Dave DeMaagd added a comment - KAFKA-733 is already out there for the assembly stuff, though there was some question about the conflict handling.
        Hide
        David Arthur added a comment -

        Not sure if anything actually changed, but this works for me now. For me, it turned out to be an issue with SBT and Ivy. I cleared ~/.ivy2/cache/org.scala* and ~/.ivy2/cache/scala* and things started working again.

        Show
        David Arthur added a comment - Not sure if anything actually changed, but this works for me now. For me, it turned out to be an issue with SBT and Ivy. I cleared ~/.ivy2/cache/org.scala* and ~/.ivy2/cache/scala* and things started working again.
        Hide
        Diederik van Liere added a comment -

        According to http://kafka.apache.org/08/quickstart.html you also need to run ./sbt assembly-package-dependency

        Show
        Diederik van Liere added a comment - According to http://kafka.apache.org/08/quickstart.html you also need to run ./sbt assembly-package-dependency

          People

          • Assignee:
            Unassigned
            Reporter:
            David Arthur
          • Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Development