[BEAM-4567] Can't use mongo connector with Atlas MongoDB

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.12.0
• Component/s: io-java-mongodb
• Labels: mongodb
• Environment: Google Cloud Dataflow
• Assignee: Ahmed El.Hussaini
• Reporter: Lucas de Sio Rosa
• Created: 15/Jun/18 15:06
• Updated: 15/Nov/20 12:59
• Resolved: 27/Mar/19 20:13

Description

I can't use the MongoDB connector with a managed Atlas instance. The current implementations makes use of splitVector which is a high-privilege function that cannot be assigned to any user in Atlas.

An open Jira issue for MongoDB suggests using $sample and $bucketAuto to circunvent this necessity.

Following is the exception thrown (removed some identifiable information):
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.mongodb.MongoCommandException: Command failed with error 13: 'not authorized on <collection> to execute command { splitVector: "<collection>.<table>", keyPattern:

{ _id: 1 }

, force: false, maxChunkSize: 1 }' on server <server>. The full response is { "ok" : 0.0, "errmsg" : "not authorized on <collection> to execute command { splitVector: \"<collection>.<table>\", keyPattern:

{ _id: 1 }

, force: false, maxChunkSize: 1 }", "code" : 13, "codeName" : "Unauthorized" }
 
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
 
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
 
at br.dotz.datalake.ingest.mongodb.MongoDBCollectorPipeline.main(MongoDBCollectorPipeline.java:27)
 
Caused by: com.mongodb.MongoCommandException: Command failed with error 13: 'not authorized on <collection> to execute command { splitVector: "<collection>.<table>", keyPattern:

{ _id: 1 }

, force: false, maxChunkSize: 1 }' on server <server>. The full response is { "ok" : 0.0, "errmsg" : "not authorized on <collection> to execute command { splitVector: \"<collection>.<table>\", keyPattern:

{ _id: 1 }

, force: false, maxChunkSize: 1 }", "code" : 13, "codeName" : "Unauthorized" }
 
at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
 
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)
 
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
 
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
 
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:173)
 
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:215)
 
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:186)
 
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:178)
 
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:91)
 
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
 
at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
 
at com.mongodb.Mongo.execute(Mongo.java:772)
 
at com.mongodb.Mongo$2.execute(Mongo.java:759)
 
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
 
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
 
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
 
at org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
 
at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:210)
 
at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87)
 
at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62)

Issue Links

• is related to: BEAM-6241 Add support for aggregates using withQueryFn to MongoDbIO
• relates to: BEAM-11266 Cannot use Python MongoDB connector with Atlas MongoDB