[DRILL-3522] IllegalStateException from Mongo storage plugin

• Type: Bug
• Status: Reviewable
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: Storage - MongoDB
• Assignee: Adam Gilmore
• Reporter: Adam Gilmore
• Created: 21/Jul/15 02:40
• Updated: 22/Apr/16 00:21

Description

With a Mongo storage plugin enabled, we are sporadically getting the following exception when running queries (even not against the Mongo storage plugin):

SYSTEM ERROR: IllegalStateException: state should be: open



  (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: org.apache.drill.common.exceptions.DrillRuntimeException: state should be: open
    org.apache.drill.exec.work.foreman.Foreman.run():253
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745
  Caused By (com.google.common.util.concurrent.UncheckedExecutionException) org.apache.drill.common.exceptions.DrillRuntimeException: state should be: open
    com.google.common.cache.LocalCache$Segment.get():2263
    com.google.common.cache.LocalCache.get():4000
    com.google.common.cache.LocalCache.getOrLoad():4004
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
    org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
    org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
    org.apache.drill.exec.ops.QueryContext.getRootSchema():165
    org.apache.drill.exec.ops.QueryContext.getRootSchema():154
    org.apache.drill.exec.ops.QueryContext.getRootSchema():142
    org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
    org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
    org.apache.drill.exec.work.foreman.Foreman.runSQL():901
    org.apache.drill.exec.work.foreman.Foreman.run():242
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745
  Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) state should be: open
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():98
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
    com.google.common.cache.LocalCache$Segment.loadSync():2379
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
    com.google.common.cache.LocalCache$Segment.get():2257
    com.google.common.cache.LocalCache.get():4000
    com.google.common.cache.LocalCache.getOrLoad():4004
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
    org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
    org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
    org.apache.drill.exec.ops.QueryContext.getRootSchema():165
    org.apache.drill.exec.ops.QueryContext.getRootSchema():154
    org.apache.drill.exec.ops.QueryContext.getRootSchema():142
    org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
    org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
    org.apache.drill.exec.work.foreman.Foreman.runSQL():901
    org.apache.drill.exec.work.foreman.Foreman.run():242
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745
  Caused By (java.lang.IllegalStateException) state should be: open
    com.mongodb.assertions.Assertions.isTrue():70
    com.mongodb.connection.BaseCluster.selectServer():79
    com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():75
    com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():71
    com.mongodb.binding.ClusterBinding.getReadConnectionSource():63
    com.mongodb.operation.OperationHelper.withConnection():166
    com.mongodb.operation.ListDatabasesOperation.execute():100
    com.mongodb.operation.ListDatabasesOperation.execute():52
    com.mongodb.Mongo.execute():738
    com.mongodb.Mongo$2.execute():725
    com.mongodb.OperationIterable.iterator():47
    com.mongodb.OperationIterable.forEach():66
    com.mongodb.ListDatabasesIterableImpl.forEach():72
    com.mongodb.MappingIterable.forEach():50
    com.mongodb.MappingIterable.into():60
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():91
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
    com.google.common.cache.LocalCache$Segment.loadSync():2379
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
    com.google.common.cache.LocalCache$Segment.get():2257
    com.google.common.cache.LocalCache.get():4000
    com.google.common.cache.LocalCache.getOrLoad():4004
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
    org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
    org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
    org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
    org.apache.drill.exec.ops.QueryContext.getRootSchema():165
    org.apache.drill.exec.ops.QueryContext.getRootSchema():154
    org.apache.drill.exec.ops.QueryContext.getRootSchema():142
    org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
    org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
    org.apache.drill.exec.work.foreman.Foreman.runSQL():901
    org.apache.drill.exec.work.foreman.Foreman.run():242
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745

Upon investigation, the issue appears to be registering the Mongo schemas on connection (because the storage plugin is enabled).

Basically, it appears that recent changes when upgrading the Mongo driver meant that MongoSchemaFactory indefinitely holds onto a MongoClient, even though they are part of a cache that expire after 24 hours. This means that MongoSchemaFactory ends up trying to use a MongoClient that is closed, and thus the exception occurs.

Considering we already have a cache for MongoClient, it is safe for MongoSchemaFactory to call plugin.getClient() every time.