[KAFKA-9362] Missing GC logs when running Kafka with Java 11 with OpenJ9 VM

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.1, 2.2.2, 2.4.0, 2.3.1
• Fix Version/s: 2.1.2, 2.2.3, 2.3.2, 2.4.2
• Component/s: core
• Labels: OpenJ9
• Environment: AdoptOpenJDK11.0.5 (LTS release) on Linux, Windows, etc.
• Assignee: Unassigned
• Reporter: Alexandre Vermeerbergen
• Created: 03/Jan/20 12:09
• Updated: 02/Mar/20 20:19

Description

When running Kafka server with a Java 11 JRE based on OpenJ9 JVM (instead of usual HotSpot), we get this message in output:

$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
JVMJ9VM085W Malformed option: '-Xlog:gc*:file=/home/data/ave/Java/kafka_2.12-2.4.0/bin/../logs/zookeeper-gc.log:time'
[2020-01-03 12:46:35,575] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-01-03 12:46:35,576] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-01-03 12:46:35,577] INFO clientPortAddress is 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
...

and likewise when running 'bin/kafka-server-start.sh config/server.properties'

While this does not prevent Kafka server (and its Zookeeper server) from running, it has the effect that GC logs are not written, making diagnostics harder than when running Kafka with a Java 11 based on HotSpot JVM.

This is because OpenJ9 does not implements JEP 158, and thus even Java 11 versions based on OpenJ9 still rely on the same GC logging options as Java 8.

I believe that it is easy to fix this issue by making a change to https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh, lines 295-299:

if [[ "$JAVA_MAJOR_VERSION" -ge "9" ]] ; then
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
else
KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
fi

which could be replaced by:

JAVA_VM_ISJ9=$($JAVA -version 2>&1|grep J9)
if [[ "$JAVA_MAJOR_VERSION" -ge "9" || -n "$JAVA_VM_ISJ9" ]] ; then
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
else
KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
fi

Please let me know if that sounds good to you.

Please note that IBM AIX Java is also based on OpenJ9, so fixing this issue would also nice for AIX users !

Kind regards,
Alexandre

 

Issue Links

• links to: GitHub Pull Request #7896