[KAFKA-5363] Add ability to batch restore and receive restoration stats.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: streams
• Labels: kip
• Assignee: Bill Bejeck
• Reporter: Bill Bejeck
• Created: 01/Jun/17 20:12
• Updated: 28/Jul/17 18:32
• Resolved: 28/Jul/17 18:31

Description

Currently, when restoring a state store in a Kafka Streams application, we put one key-value at a time into the store.

This task aims to make this recovery more efficient by creating a new interface with "restoreAll" functionality allowing for bulk writes by the underlying state store implementation.

The proposal will also add "beginRestore" and "endRestore" callback methods potentially used for
Tracking when the bulk restoration process begins and ends.
Keeping track of the number of records and last offset restored.

KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-167%3A+Add+interface+for+the+state+store+restoration+process

Issue Links

• is related to: KAFKA-4322 StateRestoreCallback begin and end indication
• links to: GitHub Pull Request #3325
• mentioned in: Page Loading...

Sub-Tasks

• Optimize RocksDb config for fast recovery/bulk load: Resolved