[SAMZA-2383] Make Samza internal state (backed by RocksDB) queryable

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: container
• Assignee: Dengpan Yin
• Reporter: Dengpan Yin
• Created: 13/Nov/19 00:28
• Updated: 13/Nov/19 00:28

Description

Samza is using RocksDB as the KV store for internal state, both key and values are encoded as byte array.  Some application owners want to query the internal state when the job is running for troubleshooting.  To do so, the application owner needs to access the cluster where the container is running.  For security reasons, the application owner usually does not haver permission to access the shared cluster, which prevent the application owner from querying the RocksDB directly. 

Samza-beam-runner also include some meta information in the key, such as the window information.   The application owner usually does not know the window information when perform the query. The application owner needs to understand the encoding format in order to do a range query.

 

It will be helpful to provide a convenient interface to the user to query the state and show the results.  For instance, the user can specify the container ID, partition#, the state name, the query key and SERDE.  The request will be processed by the container and present the results to the user.