[KAFKA-17184] Remote index cache noisy logging

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Francois Visconte
• Created: 23/Jul/24 08:08
• Updated: 23/Jul/24 08:10

Description

We have a tiered storage cluster where some consumers are constantly lagging behind.

On this cluster, we get a ton of error logs and fail fetches with the following symptom:

java.lang.IllegalStateException: This entry is marked for cleanup
	at org.apache.kafka.storage.internals.log.RemoteIndexCache$Entry.lookupOffset(RemoteIndexCache.java:569)
	at org.apache.kafka.storage.internals.log.RemoteIndexCache.lookupOffset(RemoteIndexCache.java:446)
	at kafka.log.remote.RemoteLogManager.lookupPositionForOffset(RemoteLogManager.java:1445)
	at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1391)
	at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:62)
	at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:31)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

I believe this should be handled differently by either

• Set log level to warn or info
• reload the index entry synchronously when an offset is requested and the entry is marked for cleanup.

We do use the default setting for remote.log.index.file.cache.total.size.bytes (1GiB).

Issue Links

• is a child of: KAFKA-16947 Kafka Tiered Storage V2