Uploaded image for project: 'ManifoldCF'
  1. ManifoldCF
  2. CONNECTORS-1116

"Too many open files" IO exception from Rest API

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • ManifoldCF 1.7.2
    • ManifoldCF 1.8, ManifoldCF 2.0
    • API
    • None
    • Ubuntu Linux, Tomcat, Zookeeper

    Description

      Using the REST API to get a list of all the job definitions.

      Using lsof, I can see that the total number of open files jumps from around 160 to 4500 as the call is made.

      Also, I should note that there are around 4000 jobs under MCF control.

      Exception below:

      ERROR 2014-11-26 12:46:07,042 (http-bio-8080-exec-9) - Zookeeper initialization error: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
      org.apache.manifoldcf.core.interfaces.ManifoldCFException: Zookeeper initialization error: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
      at org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:82)
      at org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.<init>(ZooKeeperConnection.java:66)
      at org.apache.manifoldcf.core.lockmanager.ZooKeeperConnectionPool.grab(ZooKeeperConnectionPool.java:48)
      at org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.obtainGlobalReadLock(ZooKeeperLockObject.java:190)
      at org.apache.manifoldcf.core.lockmanager.LockObject.enterReadLock(LockObject.java:310)
      at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:261)
      at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
      at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
      at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:299)
      at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
      at org.apache.manifoldcf.crawler.jobs.Jobs.loadMultiple(Jobs.java:926)
      at org.apache.manifoldcf.crawler.jobs.Jobs.getAll(Jobs.java:774)
      at org.apache.manifoldcf.crawler.jobs.JobManager.getAllJobs(JobManager.java:760)
      at org.apache.manifoldcf.crawler.system.ManifoldCF.apiReadJobs(ManifoldCF.java:1299)
      at org.apache.manifoldcf.crawler.system.ManifoldCF.executeReadCommand(ManifoldCF.java:2919)
      at org.apache.manifoldcf.apiservlet.APIServlet.executeRead(APIServlet.java:232)
      at org.apache.manifoldcf.apiservlet.APIServlet.doGet(APIServlet.java:78)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
      at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1778)
      at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
      at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
      at org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74)
      ... 34 more
      Caused by: java.io.IOException: Too many open files
      at sun.nio.ch.IOUtil.makePipe(Native Method)
      at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
      at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
      at java.nio.channels.Selector.open(Selector.java:227)
      at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
      at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      at java.lang.Class.newInstance(Class.java:374)
      at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1775)
      ... 37 more

      Attachments

        1. CONNECTORS-1116-1x.patch
          94 kB
          Karl Wright
        2. CONNECTORS-1116.patch
          21 kB
          Karl Wright
        3. CONNECTORS-1116.2.patch
          93 kB
          Karl Wright

        Activity

          People

            kwright@metacarta.com Karl Wright
            moleski Adrian Conlon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: