Major Description
Currently PhoenixIndexFailurePolicy, which is the default policy, delegates to KillServerOnFailurePolicy. This is hardcoded in the constructor. Apparently this was added for a specific use case, BLOCK_DATA_TABLE_WRITES_ON_WRITE_FAILURE, and the policy itself derives from the days where forcing a RS kill was in effect the way to 'rebuild' the index via WAL replay.
There are still cases where it's applicable, such as when Syscat itself cannot be updated in order to e.g. disable an index. However, killing the RS may be too aggressive for some, who might prefer a temporarily out of sync index to a potentially cascading wave of aborts.
We should add a config option to control this.