Solr
  1. Solr
  2. SOLR-2308

Race condition still exists in StreamingUpdateSolrServer which could cause it to hang

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.1
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: clients - java
    • Labels:
      None

      Description

      We are still seeing the same issue as SOLR-1711 & SOLR-1885 with Solr1.4.1
      We get into this situation when all the runner threads die due to a broken pipe, while the BlockingQueue is still full. All of the producer threads are all blocked on the BlockingQueue.put() method. Since the runners are spawned by the producers, which are all blocked, runner threads never get created to drain the queue.

      Here's a potential fix. In the runner code, replace these lines:

      // remove it from the list of running things...
      synchronized (runners) {
          runners.remove( this );
      }
      

      with these lines:

      // remove it from the list of running things unless we are the last runner and the queue is full...
      synchronized (runners) {
          if (runners.size() == 1 && queue.remainingCapacity() == 0) {
              // keep this runner alive
              scheduler.execute(this);
          } else {
              runners.remove( this );
          }
      }
      

        Issue Links

          Activity

          Jan Høydahl made changes -
          Link This issue duplicates SOLR-1711 [ SOLR-1711 ]
          Grant Ingersoll made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Yonik Seeley made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 3.1 [ 12314371 ]
          Fix Version/s 4.0 [ 12314992 ]
          Resolution Fixed [ 1 ]
          Johannes made changes -
          Description We are still seeing the same issue as SOLR-1711 & SOLR-1885 with Solr1.4.1
          We get into this situation when all the runner threads die due to a broken pipe, while the BlockingQueue is still full. All of the producer threads are all blocked on the BlockingQueue.put() method. Since the runners are spawned by the producers, which are all blocked, runner threads never get created to drain the queue.

          Here's a potential fix. In the runner code, replace these lines:

          {code}
          // remove it from the list of running things...
          synchronized (runners) { runners.remove( this ); }
          {code}

          with these lines:

          {code}
          // remove it from the list of running things unless we are the last runner and the queue is full...
          synchronized (runners) {
          if (runners.size() == 1 && queue.remainingCapacity() == 0) { // keep this runner alive scheduler.execute(this); } else { runners.remove( this ); }
          }
          {code}
          We are still seeing the same issue as SOLR-1711 & SOLR-1885 with Solr1.4.1
          We get into this situation when all the runner threads die due to a broken pipe, while the BlockingQueue is still full. All of the producer threads are all blocked on the BlockingQueue.put() method. Since the runners are spawned by the producers, which are all blocked, runner threads never get created to drain the queue.

          Here's a potential fix. In the runner code, replace these lines:

          {code}
          // remove it from the list of running things...
          synchronized (runners) {
              runners.remove( this );
          }
          {code}

          with these lines:

          {code}
          // remove it from the list of running things unless we are the last runner and the queue is full...
          synchronized (runners) {
              if (runners.size() == 1 && queue.remainingCapacity() == 0) {
                  // keep this runner alive
                  scheduler.execute(this);
              } else {
                  runners.remove( this );
              }
          }
          {code}
          Johannes made changes -
          Field Original Value New Value
          Description We are still seeing the same issue as SOLR-1711 & SOLR-1885 with Solr1.4.1
          We get into this situation when all the runner threads die due to a broken pipe, while the BlockingQueue is still full. All of the producer threads are all blocked on the BlockingQueue.put() method. Since the runners are spawned by the producers, which are all blocked, runner threads never get created to drain the queue.

          Here's a potential fix. In the runner code, replace these lines:

          // remove it from the list of running things...
          synchronized (runners) { runners.remove( this ); }

          with these lines:

          // remove it from the list of running things unless we are the last runner and the queue is full...
          synchronized (runners) {
          if (runners.size() == 1 && queue.remainingCapacity() == 0) { // keep this runner alive scheduler.execute(this); } else { runners.remove( this ); }
          }
          We are still seeing the same issue as SOLR-1711 & SOLR-1885 with Solr1.4.1
          We get into this situation when all the runner threads die due to a broken pipe, while the BlockingQueue is still full. All of the producer threads are all blocked on the BlockingQueue.put() method. Since the runners are spawned by the producers, which are all blocked, runner threads never get created to drain the queue.

          Here's a potential fix. In the runner code, replace these lines:

          {code}
          // remove it from the list of running things...
          synchronized (runners) { runners.remove( this ); }
          {code}

          with these lines:

          {code}
          // remove it from the list of running things unless we are the last runner and the queue is full...
          synchronized (runners) {
          if (runners.size() == 1 && queue.remainingCapacity() == 0) { // keep this runner alive scheduler.execute(this); } else { runners.remove( this ); }
          }
          {code}
          Johannes created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Johannes
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development