Details
-
Improvement
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
1.6.2, 1.7.0
-
None
-
Patch
Description
When IntervalJoin, it is very slow to get left or right buffer's entries. Because we have to scan all buffer's values, including the deleted values which are out of time range. These deleted values's processing consumes too much time in RocksDB's level 0. Since lowerBound is known, it can be optimized by seek from the timestamp of lowerBound.
Our usage is like below:
labelStream.keyBy(uuid).intervalJoin(adLogStream.keyBy(uuid))
.between(Time.milliseconds(0), Time.milliseconds(600000))
.process(new processFunction())
.sink(kafkaProducer)
Our data is huge. The job always runs for an hour and is stuck by RocksDB's seek when get buffer's entries. We use rocksDB's data to simulate the problem RocksDB and find that it takes too much time in deleted values. So we decide to optimize it by assigning the lowerBound instead of global search.
Attachments
Issue Links
- is related to
-
FLINK-16392 oneside sorted cache in intervaljoin
- Closed
- links to