Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7113

Multiple calls to UpdateLog#init is not thread safe with respect to the HDFS FileSystem client object usage.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      I notice this issue while trying to do some heavy indexing into Solr. (700K docs per minute)

      Solr log errors

      15:42:47
      ERROR
      HdfsTransactionLog
      Exception closing tlog.
      java.io.IOException: Filesystem closed
      	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:765)
      	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1898)
      	at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1859)
      	at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
      	at org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:303)
      	at org.apache.solr.update.TransactionLog.decref(TransactionLog.java:504)
      	at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:335)
      	at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:628)
      	at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:600)
      	at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      15:42:47
      ERROR
      CommitTracker
      auto commit error...:org.apache.solr.common.SolrException: java.io.IOException: Filesystem closed
      auto commit error...:org.apache.solr.common.SolrException: java.io.IOException: Filesystem closed
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                markrmiller@gmail.com Mark Miller
                Reporter:
                vamsee Vamsee Yarlagadda
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: