Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6110

DIH increase tcp connections so full-import failed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 4.7.2
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Debian Lenny

      Description

      Before i used solr 4.3.1 and i had no problem. Since i use solr 4.7.2 i have a problem with the DIH.
      When i do a full import with DIH the number of TCP connection increase.
      For example i have 240000 documents to index and the full-import stop randomly between 6000 and 9000 docs indexed with the error :
      com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

      With the command (while full-import is running) :
      netstat | grep localdo

      I saw that there are many connections in status "TIME_WAIT"
      The problem is that the OS don't free those connections quickly. So the tcp connections queue increase to its limit and crash ---> full-import failed.

      To solve this problem i found a solution :
      If i change OS settings "/proc/sys/net/ipv4/tcp_tw_reuse" to 1 and "/proc/sys/net/ipv4/tcp_tw_recycle" to 1 my full-import works to the end.

      So why connections are not closed in this new version ?
      And why the java socket option SO_REUSEADDR is not used ?

      I'm not a Java developer so maybe i miss something but the only way for me to do a success full-import was to use the solution above.

      Thank you by advance for your answers to help me solve this problem with a better solution than mine.

      Thomas

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tom42 Gambier
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: