Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9290

TCP-connections in CLOSE_WAIT spike during heavy indexing and do not decrease

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.3.2, 5.4.1, 5.5.1, 5.5.2, 6.0, 6.0.1, 6.1
    • Fix Version/s: 5.5.3, 6.2
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Heavy indexing on Solr with SSL leads to a lot of connections in CLOSE_WAIT state.

      At my workplace, we have seen this issue only with 5.5.1 and could not reproduce it with 5.4.1 but from my conversation with Shalin, he knows of users with 5.3.1 running into this issue too.

      Here's an excerpt from the email Shai Erera sent to the mailing list (about what we see:

      1) It consistently reproduces on 5.5.1, but does not reproduce on 5.4.1
      2) It does not reproduce when SSL is disabled
      3) Restarting the Solr process (sometimes both need to be restarted), the
      count drops to 0, but if indexing continues, they climb up again

      When it does happen, Solr seems stuck. The leader cannot talk to the
      replica, or vice versa, the replica is usually put in DOWN state and
      there's no way to fix it besides restarting the JVM.

      Here's the mail thread: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201607.mbox/%3C46cc66220a8143dc903fa34e792059c4@vp-exc01.dips.local%3E

      Creating this issue so we could track this and have more people comment on what they see.

        Attachments

        1. setup-solr.sh
          0.6 kB
          Shalin Shekhar Mangar
        2. SOLR-9290-debug.patch
          15 kB
          Shalin Shekhar Mangar
        3. SOLR-9290-debug.patch
          17 kB
          Shalin Shekhar Mangar
        4. setup-solr.sh
          1 kB
          Hoss Man
        5. index.sh
          0.8 kB
          Hoss Man
        6. SOLR-9290.patch
          17 kB
          Shalin Shekhar Mangar
        7. SOLR-9290.patch
          17 kB
          Shalin Shekhar Mangar
        8. SOLR-9290.patch
          19 kB
          Shalin Shekhar Mangar

          Activity

            People

            • Assignee:
              shalinmangar Shalin Shekhar Mangar
              Reporter:
              anshumg Anshum Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: