Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9362

Missing GC logs when running Kafka with Java 11 with OpenJ9 VM

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.1, 2.2.2, 2.4.0, 2.3.1
    • Fix Version/s: 2.1.2, 2.2.3, 2.3.2, 2.4.2
    • Component/s: core
    • Labels:
    • Environment:
      AdoptOpenJDK11.0.5 (LTS release) on Linux, Windows, etc.

      Description

      When running Kafka server with a Java 11 JRE based on OpenJ9 JVM (instead of usual HotSpot), we get this message in output:

      $ ./bin/zookeeper-server-start.sh config/zookeeper.properties
      JVMJ9VM085W Malformed option: '-Xlog:gc*:file=/home/data/ave/Java/kafka_2.12-2.4.0/bin/../logs/zookeeper-gc.log:time'
      [2020-01-03 12:46:35,575] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
      [2020-01-03 12:46:35,576] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
      [2020-01-03 12:46:35,577] INFO clientPortAddress is 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
      ...

      and likewise when running 'bin/kafka-server-start.sh config/server.properties'

      While this does not prevent Kafka server (and its Zookeeper server) from running, it has the effect that GC logs are not written, making diagnostics harder than when running Kafka with a Java 11 based on HotSpot JVM.

      This is because OpenJ9 does not implements JEP 158, and thus even Java 11 versions based on OpenJ9 still rely on the same GC logging options as Java 8.

      I believe that it is easy to fix this issue by making a change to https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh, lines 295-299:

      if [[ "$JAVA_MAJOR_VERSION" -ge "9" ]] ; then
      KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
      else
      KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
      fi

      which could be replaced by:

      JAVA_VM_ISJ9=$($JAVA -version 2>&1|grep J9)
      if [[ "$JAVA_MAJOR_VERSION" -ge "9" || -n "$JAVA_VM_ISJ9" ]] ; then
      KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
      else
      KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
      fi

      Please let me know if that sounds good to you.

      Please note that IBM AIX Java is also based on OpenJ9, so fixing this issue would also nice for AIX users !

      Kind regards,
      Alexandre

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                avermeerbergen Alexandre Vermeerbergen
              • Votes:
                3 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: