Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4893

async topic deletion conflicts with max topic length

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0, 2.1.2, 2.2.2
    • Component/s: log
    • Labels:
      None

      Description

      As per the documentation, topics can be only 249 characters long to line up with typical filesystem limitations:

      Each sharded partition log is placed into its own folder under the Kafka log directory. The name of such folders consists of the topic name, appended by a dash (-) and the partition id. Since a typical folder name can not be over 255 characters long, there will be a limitation on the length of topic names. We assume the number of partitions will not ever be above 100,000. Therefore, topic names cannot be longer than 249 characters. This leaves just enough room in the folder name for a dash and a potentially 5 digit long partition id.

      kafka.common.Topic.maxNameLength is set to 249 and is used during validation.

      This limit ends up not being quite right since topic deletion ends up renaming the directory to the form topic-partition.uniqueId-delete as can be seen in LogManager.asyncDelete:

      val dirName = new StringBuilder(removedLog.name)
                        .append(".")
                        .append(java.util.UUID.randomUUID.toString.replaceAll("-",""))
                        .append(Log.DeleteDirSuffix)
                        .toString()
      

      So the unique id and "-delete" suffix end up hogging some of the characters. Deleting a long-named topic results in a log message such as the following:

      kafka.common.KafkaStorageException: Failed to rename log directory from /tmp/kafka-logs0/000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000-0 to /tmp/kafka-logs0/000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000-0.797bba3fb2464729840f87769243edbb-delete
        at kafka.log.LogManager.asyncDelete(LogManager.scala:439)
        at kafka.cluster.Partition$$anonfun$delete$1.apply$mcV$sp(Partition.scala:142)
        at kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:137)
        at kafka.cluster.Partition$$anonfun$delete$1.apply(Partition.scala:137)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
        at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:221)
        at kafka.cluster.Partition.delete(Partition.scala:137)
        at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:230)
        at kafka.server.ReplicaManager$$anonfun$stopReplicas$2.apply(ReplicaManager.scala:260)
        at kafka.server.ReplicaManager$$anonfun$stopReplicas$2.apply(ReplicaManager.scala:259)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:259)
        at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:174)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:86)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:64)
        at java.lang.Thread.run(Thread.java:745)
      

      The topic after this point still exists but has Leader set to -1 and the controller recognizes the topic completion as incomplete (the topic znode is still in /admin/delete_topics).

      I don't believe linkedin has any topic name this long but I'm making the ticket in case anyone runs into this problem.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin McCabe
                Reporter:
                onurkaraman Onur Karaman
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: