Major Description
Issue is related to Region Server being killed when one region is closing and another region is trying to write index updates.
When the data table region closes it will close region server level cached/shared connections and it could interrupt other region index/index-state update.
– Region1: Closing
TrackingParallellWriterIndexCommitter#stop() {
this.retryingFactory.shutdown();
this.noRetriesFactory.shutdown();
}
closes the cached connections calling CoprocessorHConnectionTableFactory#shutdown() in ServerUtil.java
--Region2: Writing index updates
Index updates fail as connections are closed, which leads to RejectedExecutionException/Connection being null. This triggers PhoenixIndexFailurePolicy#handleFailureWithExceptions that tries to get the the syscat table using the cached connections. Here it will not be able to reach to SYSCAT , so we will trigger KillServreFailurePolicy.
CoprocessorHConnectionTableFactory#getTable()
if (connection == null || connection.isClosed()) { throw new IllegalArgumentException("Connection is null or closed."); }