Major Description
It appears that we only give the illusion of full customizability when it comes to the state stores of a windowed join. This arose due to an optimization for the performance of the spurious results fix, and means that these joins now come with one additional, and possibly unexpected, state store:
final StoreBuilder<KeyValueStore<TimestampedKeyAndJoinSide<K>, LeftOrRightValue<V1, V2>>> builder = new ListValueStoreBuilder<>( |--[ persistent ? this--> | Stores.persistentKeyValueStore(storeName) : |--[ Stores.inMemoryKeyValueStore(storeName), timestampedKeyAndJoinSideSerde, leftOrRightValueSerde, Time.SYSTEM );
where persistent is defined above that as
final boolean persistent = streamJoinedInternal.thisStoreSupplier() == null || streamJoinedInternal.thisStoreSupplier().get().persistent();
This means regardless of whether a custom state store implementation was passed in to the join, we will still insert one of our RocksDB or InMemory state stores. Which might be very surprising since the API makes it seem like the underlying stores are fully configurable.
I'm adding a warning line for this in PR #13682 but we should really make this hidden state store fully configurable like the window stores currently are (which will require a KIP)