The existing config override is sufficient for "connection refused" errors. It doesn't cover "connection timeout" errors, which is configured separately in the base RPC client code.
After the AM exits, we would expect connection attempts to cause an immediate connection refused error, not a longer connection timeout error. After all, the packets can get to their destination. There's just no server listening anymore. The reason I saw connection timeouts was a side effect of a feature of Windows Firewall called Stealth Mode. This feature is on by default, and it intentionally drops outbound TCP RST packets for connections initiated against a port with no server listening.
Without getting the RST, the client doesn't know that a connection has been refused, and so it just has to wait for the longer timeout condition. It's possible to disable stealth mode by setting a registry key and restarting the firewall:
That article might be out of date though, because I found that this registry key was really at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\SharedAccess\Parameters\FirewallPolicy\PublicProfile in my environments.
My only known repro right now is on Windows. I'm leaving this information here for anyone who might notice similar problems on other RPC interactions. I'd still like to get a configuration patch into the client for this.