Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3522

IllegalStateException from Mongo storage plugin

    Details

    • Type: Bug
    • Status: Reviewable
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: Storage - MongoDB
    • Labels:
      None

      Description

      With a Mongo storage plugin enabled, we are sporadically getting the following exception when running queries (even not against the Mongo storage plugin):

      SYSTEM ERROR: IllegalStateException: state should be: open
      
      
      
        (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: org.apache.drill.common.exceptions.DrillRuntimeException: state should be: open
          org.apache.drill.exec.work.foreman.Foreman.run():253
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (com.google.common.util.concurrent.UncheckedExecutionException) org.apache.drill.common.exceptions.DrillRuntimeException: state should be: open
          com.google.common.cache.LocalCache$Segment.get():2263
          com.google.common.cache.LocalCache.get():4000
          com.google.common.cache.LocalCache.getOrLoad():4004
          com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
          org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
          org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
          org.apache.drill.exec.ops.QueryContext.getRootSchema():165
          org.apache.drill.exec.ops.QueryContext.getRootSchema():154
          org.apache.drill.exec.ops.QueryContext.getRootSchema():142
          org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
          org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
          org.apache.drill.exec.work.foreman.Foreman.runSQL():901
          org.apache.drill.exec.work.foreman.Foreman.run():242
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) state should be: open
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():98
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
          com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
          com.google.common.cache.LocalCache$Segment.loadSync():2379
          com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
          com.google.common.cache.LocalCache$Segment.get():2257
          com.google.common.cache.LocalCache.get():4000
          com.google.common.cache.LocalCache.getOrLoad():4004
          com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
          org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
          org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
          org.apache.drill.exec.ops.QueryContext.getRootSchema():165
          org.apache.drill.exec.ops.QueryContext.getRootSchema():154
          org.apache.drill.exec.ops.QueryContext.getRootSchema():142
          org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
          org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
          org.apache.drill.exec.work.foreman.Foreman.runSQL():901
          org.apache.drill.exec.work.foreman.Foreman.run():242
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (java.lang.IllegalStateException) state should be: open
          com.mongodb.assertions.Assertions.isTrue():70
          com.mongodb.connection.BaseCluster.selectServer():79
          com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():75
          com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():71
          com.mongodb.binding.ClusterBinding.getReadConnectionSource():63
          com.mongodb.operation.OperationHelper.withConnection():166
          com.mongodb.operation.ListDatabasesOperation.execute():100
          com.mongodb.operation.ListDatabasesOperation.execute():52
          com.mongodb.Mongo.execute():738
          com.mongodb.Mongo$2.execute():725
          com.mongodb.OperationIterable.iterator():47
          com.mongodb.OperationIterable.forEach():66
          com.mongodb.ListDatabasesIterableImpl.forEach():72
          com.mongodb.MappingIterable.forEach():50
          com.mongodb.MappingIterable.into():60
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():91
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
          com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
          com.google.common.cache.LocalCache$Segment.loadSync():2379
          com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
          com.google.common.cache.LocalCache$Segment.get():2257
          com.google.common.cache.LocalCache.get():4000
          com.google.common.cache.LocalCache.getOrLoad():4004
          com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
          org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
          org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
          org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
          org.apache.drill.exec.ops.QueryContext.getRootSchema():165
          org.apache.drill.exec.ops.QueryContext.getRootSchema():154
          org.apache.drill.exec.ops.QueryContext.getRootSchema():142
          org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
          org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
          org.apache.drill.exec.work.foreman.Foreman.runSQL():901
          org.apache.drill.exec.work.foreman.Foreman.run():242
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
      

      Upon investigation, the issue appears to be registering the Mongo schemas on connection (because the storage plugin is enabled).

      Basically, it appears that recent changes when upgrading the Mongo driver meant that MongoSchemaFactory indefinitely holds onto a MongoClient, even though they are part of a cache that expire after 24 hours. This means that MongoSchemaFactory ends up trying to use a MongoClient that is closed, and thus the exception occurs.

      Considering we already have a cache for MongoClient, it is safe for MongoSchemaFactory to call plugin.getClient() every time.

        Attachments

          Activity

            People

            • Assignee:
              dragoncurve Adam Gilmore
              Reporter:
              dragoncurve Adam Gilmore
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: