Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1323

log.dirs server property no longer supports relative directories

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 0.8.1.1
    • None
    • None

    Description

      This seems to have been caused by KAFKA-1315 - we now don't support relative directories.

      Steps to reproduce:

      • Set a relative directory for log.dirs. E.g., log.dirs=data/kafka-logs
      • Bring up the broker and produce some messages: ./bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 1000 --topic test

      Attachments

        1. kafka-1323-trunk-test-failures.png
          86 kB
          Neha Narkhede
        2. KAFKA-1323.patch
          7 kB
          Timothy Chen
        3. KAFKA-1323.patch
          14 kB
          Guozhang Wang
        4. KAFKA-1323.patch
          60 kB
          Guozhang Wang
        5. KAFKA-1323.patch
          12 kB
          Guozhang Wang
        6. KAFKA-1323_2014-04-16_12:05:40.patch
          16 kB
          Guozhang Wang
        7. KAFKA-1323_2014-04-16_11:53:39.patch
          15 kB
          Guozhang Wang
        8. KAFKA-1323_2014-04-14_16:13:07.patch
          14 kB
          Guozhang Wang
        9. KAFKA-1323_2014-04-02_12:17:57.patch
          19 kB
          Timothy Chen
        10. KAFKA-1323_2014-03-31_11:57:37.patch
          15 kB
          Timothy Chen
        11. KAFKA-1323_2014-03-27_16:17:26.patch
          78 kB
          Timothy Chen

        Issue Links

          Activity

            java.util.NoSuchElementException: key not found: data/kafka-logs
                    at scala.collection.MapLike$class.default(MapLike.scala:223)
                    at scala.collection.immutable.Map$Map1.default(Map.scala:93)
                    at scala.collection.MapLike$class.apply(MapLike.scala:134)
                    at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
                    at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:93)
                    at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:178)
                    at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:178)
                    at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
                    at kafka.cluster.Partition.makeLeader(Partition.scala:178)
                    at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:309)
                    at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:308)
                    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
                    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
                    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
                    at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
                    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
                    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
                    at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
                    at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:308)
                    at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:260)
                    at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:100)
                    at kafka.server.KafkaApis.handle(KafkaApis.scala:71)
                    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
                    at java.lang.Thread.run(Thread.java:662)
            

            I think we just need to update the look-up key in getOrCreateReplica with
            the absolute path, but we should check if there are other accesses to the
            same map which needs an update.

            jjkoshy Joel Jacob Koshy added a comment - java.util.NoSuchElementException: key not found: data/kafka-logs at scala.collection.MapLike$ class. default (MapLike.scala:223) at scala.collection.immutable.Map$Map1. default (Map.scala:93) at scala.collection.MapLike$ class. apply(MapLike.scala:134) at scala.collection.immutable.Map$Map1.apply(Map.scala:93) at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:93) at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:178) at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:178) at scala.collection.immutable.Set$Set1.foreach(Set.scala:81) at kafka.cluster.Partition.makeLeader(Partition.scala:178) at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:309) at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:308) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) at scala.collection.Iterator$ class. foreach(Iterator.scala:631) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) at scala.collection.mutable.HashTable$ class. foreachEntry(HashTable.scala:194) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:308) at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:260) at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:100) at kafka.server.KafkaApis.handle(KafkaApis.scala:71) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) at java.lang. Thread .run( Thread .java:662) I think we just need to update the look-up key in getOrCreateReplica with the absolute path, but we should check if there are other accesses to the same map which needs an update.
            nehanarkhede Neha Narkhede added a comment -

            oops, this is my bad. Missed this during the review. Which points out that we need a unit test to cover this as well.

            nehanarkhede Neha Narkhede added a comment - oops, this is my bad. Missed this during the review. Which points out that we need a unit test to cover this as well.
            tnachen Timothy Chen added a comment -

            Created reviewboard https://reviews.apache.org/r/19626/
            against branch origin/trunk

            tnachen Timothy Chen added a comment - Created reviewboard https://reviews.apache.org/r/19626/ against branch origin/trunk
            tnachen Timothy Chen added a comment -

            Updated reviewboard https://reviews.apache.org/r/19626/
            against branch origin/trunk

            tnachen Timothy Chen added a comment - Updated reviewboard https://reviews.apache.org/r/19626/ against branch origin/trunk
            nehanarkhede Neha Narkhede added a comment -

            I ran into quite a few unit test failures with the patch on trunk -

            nehanarkhede Neha Narkhede added a comment - I ran into quite a few unit test failures with the patch on trunk -
            tnachen Timothy Chen added a comment -

            Updated reviewboard https://reviews.apache.org/r/19626/
            against branch origin/trunk

            tnachen Timothy Chen added a comment - Updated reviewboard https://reviews.apache.org/r/19626/ against branch origin/trunk
            nehanarkhede Neha Narkhede added a comment -

            The tests seem to pass now. I'm a +1 once junrao's comments are addressed.

            nehanarkhede Neha Narkhede added a comment - The tests seem to pass now. I'm a +1 once junrao 's comments are addressed.
            tnachen Timothy Chen added a comment -

            Updated reviewboard https://reviews.apache.org/r/19626/
            against branch origin/trunk

            tnachen Timothy Chen added a comment - Updated reviewboard https://reviews.apache.org/r/19626/ against branch origin/trunk
            nehanarkhede Neha Narkhede added a comment -

            The same tests (mentioned above) fail for me on trunk with the latest patch.

            nehanarkhede Neha Narkhede added a comment - The same tests (mentioned above) fail for me on trunk with the latest patch.
            nehanarkhede Neha Narkhede added a comment -

            Hey tnachen, I think the trunk patch needs to be rebased.

            nehanarkhede Neha Narkhede added a comment - Hey tnachen , I think the trunk patch needs to be rebased.
            guozhang Guozhang Wang added a comment -

            Updated reviewboard against branch origin/trunk

            guozhang Guozhang Wang added a comment - Updated reviewboard against branch origin/trunk
            guozhang Guozhang Wang added a comment -

            Created reviewboard https://reviews.apache.org/r/20370/
            against branch origin/trunk

            guozhang Guozhang Wang added a comment - Created reviewboard https://reviews.apache.org/r/20370/ against branch origin/trunk
            guozhang Guozhang Wang added a comment -

            Created reviewboard https://reviews.apache.org/r/20392/
            against branch origin/0.8.1

            guozhang Guozhang Wang added a comment - Created reviewboard https://reviews.apache.org/r/20392/ against branch origin/0.8.1
            guozhang Guozhang Wang added a comment -

            Created reviewboard https://reviews.apache.org/r/20393/
            against branch origin/0.8.1

            guozhang Guozhang Wang added a comment - Created reviewboard https://reviews.apache.org/r/20393/ against branch origin/0.8.1
            guozhang Guozhang Wang added a comment -

            Please ignore the RB 20392, that was a messed one.

            guozhang Guozhang Wang added a comment - Please ignore the RB 20392, that was a messed one.

            guozhang You can mark that RB as discarded

            jjkoshy Joel Jacob Koshy added a comment - guozhang You can mark that RB as discarded
            guozhang Guozhang Wang added a comment -

            Thanks Joel. Done.

            guozhang Guozhang Wang added a comment - Thanks Joel. Done.
            guozhang Guozhang Wang added a comment -

            Updated reviewboard https://reviews.apache.org/r/20393/
            against branch origin/0.8.1

            guozhang Guozhang Wang added a comment - Updated reviewboard https://reviews.apache.org/r/20393/ against branch origin/0.8.1
            guozhang Guozhang Wang added a comment -

            Updated reviewboard https://reviews.apache.org/r/20393/
            against branch origin/0.8.1

            guozhang Guozhang Wang added a comment - Updated reviewboard https://reviews.apache.org/r/20393/ against branch origin/0.8.1

            Thanks for the patch. Committed to trunk and 0.8.1.

            jjkoshy Joel Jacob Koshy added a comment - Thanks for the patch. Committed to trunk and 0.8.1.

            People

              Unassigned Unassigned
              jjkoshy Joel Jacob Koshy
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: