[SOLR-9290] TCP-connections in CLOSE_WAIT spike during heavy indexing and do not decrease

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 5.3.2, 5.4.1, 5.5.1, 5.5.2, 6.0, 6.0.1, 6.1
• Fix Version/s: 5.5.3, 6.2
• Assignee: Shalin Shekhar Mangar
• Reporter: Anshum Gupta
• Created: 07/Jul/16 16:02
• Updated: 21/Nov/19 00:41
• Resolved: 15/Jul/16 19:30

Description

Heavy indexing on Solr with SSL leads to a lot of connections in CLOSE_WAIT state.

At my workplace, we have seen this issue only with 5.5.1 and could not reproduce it with 5.4.1 but from my conversation with Shalin, he knows of users with 5.3.1 running into this issue too.

Here's an excerpt from the email shaie sent to the mailing list (about what we see:

1) It consistently reproduces on 5.5.1, but does not reproduce on 5.4.1
2) It does not reproduce when SSL is disabled
3) Restarting the Solr process (sometimes both need to be restarted), the
count drops to 0, but if indexing continues, they climb up again

When it does happen, Solr seems stuck. The leader cannot talk to the
replica, or vice versa, the replica is usually put in DOWN state and
there's no way to fix it besides restarting the JVM.

Here's the mail thread: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201607.mbox/[e-mail]%3E

Creating this issue so we could track this and have more people comment on what they see.