It turns out that if you set storm.local.hostname on your supervisors that the netty caching code might not work. The issue is that when we go to add a netty connection to the cache we use the host name provided by the scheduling. Which ultimately comes from the storm.local.hostname setting on each of the nodes. But when we go to remove it from the cache, we use the resolved INetSocket address for the destination. If the two do not match exactly then we can close a connection, but not have it removed from the cache, so when we go to try and use it again, the connection is closed.