Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4293

ByteBufferMessageSet.deepIterator burns CPU catching EOFExceptions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.10.0.1
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      around line 110:

      try {
          while (true)
              innerMessageAndOffsets.add(readMessageFromStream(compressed))
      } catch {
          case eofe: EOFException =>
          // we don't do anything at all here, because the finally
          // will close the compressed input stream, and we simply
          // want to return the innerMessageAndOffsets
      

      the only indication the code has that the end of the oteration was reached is by catching EOFException (which will be thrown inside readMessageFromStream()).

      profiling runs performed at linkedIn show 10% of the total broker CPU time taken up by Throwable.fillInStack() because of this behaviour.

      unfortunately InputStream.available() cannot be relied upon (concrete example - GZipInputStream will not correctly return 0) so the fix would probably be a wire format change to also encode the number of messages.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                radai radai rosenblatt
                Reporter:
                radai radai rosenblatt
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: