[KAFKA-9603] Number of open files keeps increasing in Streams application

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.2.0, 2.4.0, 2.3.1
• Fix Version/s: 2.6.0
• Component/s: streams
• Environment: Spring Boot 2.2.4, OpenJDK 13, Centos image
• Assignee: Bruno Cadonna
• Reporter: Bruno Iljazovic
• Created: 25/Feb/20 09:00
• Updated: 09/Mar/21 09:12
• Resolved: 20/May/20 22:11

Description

Problem appeared when upgrading from 2.0.1 to 2.3.1.

Relevant Kafka Streams code:

KStream<String, Event1> events1 =
    builder.stream(FIRST_TOPIC_NAME, Consumed.with(stringSerde, event1Serde, event1TimestampExtractor(), null))
           .mapValues(...);        

KStream<String, Event2> events2 =
    builder.stream(SECOND_TOPIC_NAME, Consumed.with(stringSerde, event2Serde, event2TimestampExtractor(), null))
           .mapValues(...);        

var joinWindows = JoinWindows.of(Duration.of(1, MINUTES).toMillis())
                             .until(Duration.of(1, HOURS).toMillis());

events2.join(events1, this::join, joinWindows, Joined.with(stringSerde, event2Serde, event1Serde))
               .foreach(...);

Number of open *.sst files keeps increasing until eventually it hits the os limit (65536) and causes this exception:

Caused by: org.rocksdb.RocksDBException: While open a file for appending: /.../0_8/KSTREAM-JOINOTHER-0000000010-store/KSTREAM-JOINOTHER-0000000010-store.1579435200000/001354.sst: Too many open files
	at org.rocksdb.RocksDB.flush(Native Method)
	at org.rocksdb.RocksDB.flush(RocksDB.java:2394)

Here are example files that are opened and never closed:

/.../0_27/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1582459200000/000114.sst
/.../0_27/KSTREAM-JOINOTHER-0000000010-store/KSTREAM-JOINOTHER-0000000010-store.1582459200000/000065.sst
/.../0_29/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1582156800000/000115.sst
/.../0_29/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1582459200000/000112.sst
/.../0_31/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1581854400000/000051.sst

Issue Links

• links to: GitHub Pull Request #8661