Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7113

Multiple calls to UpdateLog#init is not thread safe with respect to the HDFS FileSystem client object usage.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.1, 6.0
    • None
    • None

    Description

      I notice this issue while trying to do some heavy indexing into Solr. (700K docs per minute)

      Solr log errors

      15:42:47
      ERROR
      HdfsTransactionLog
      Exception closing tlog.
      java.io.IOException: Filesystem closed
      	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:765)
      	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1898)
      	at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1859)
      	at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
      	at org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:303)
      	at org.apache.solr.update.TransactionLog.decref(TransactionLog.java:504)
      	at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:335)
      	at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:628)
      	at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:600)
      	at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      15:42:47
      ERROR
      CommitTracker
      auto commit error...:org.apache.solr.common.SolrException: java.io.IOException: Filesystem closed
      auto commit error...:org.apache.solr.common.SolrException: java.io.IOException: Filesystem closed
      

      Attachments

        1. SOLR-7113.patch
          8 kB
          Mark Miller

        Issue Links

          Activity

            People

              markrmiller@gmail.com Mark Miller
              vamsee Vamsee K. Yarlagadda
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: