[STORM-2487] getting com.mongodb.MongoBulkWriteException while trying to save bulk messages using apache storm mongo

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.1
• Component/s: storm-core, storm-mongodb
• Environment: CentOS Linux release 7.3.1611
• Assignee: Unassigned
• Reporter: Krishnakanth
• Created: 21/Apr/17 13:36
• Updated: 21/Apr/17 13:36

Description

While trying to save bulk numbers of messages by using storm-mongo, we are getting below exception

com.mongodb.MongoBulkWriteException: Bulk write operation error on server mongoserver:27017. Write errors: [BulkWriteError{index=0, code=11000, message='E11000 duplicate key error collection: NextMDC.EMAIL index: id dup key: { : "22596
079-1260-44f1-b4df-a5857f48f22d" }', details={ }}].
at com.mongodb.connection.BulkWriteBatchCombiner.getError(BulkWriteBatchCombiner.java:176) ~[stormjar.jar:?]
at com.mongodb.connection.BulkWriteBatchCombiner.throwOnError(BulkWriteBatchCombiner.java:205) ~[stormjar.jar:?]
at com.mongodb.connection.BulkWriteBatchCombiner.getResult(BulkWriteBatchCombiner.java:146) ~[stormjar.jar:?]
at com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:190) ~[stormjar.jar:?]
at com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168) ~[stormjar.jar:?]
at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230) ~[stormjar.jar:?]
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221) ~[stormjar.jar:?]
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168) ~[stormjar.jar:?]
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74) ~[stormjar.jar:?]
at com.mongodb.Mongo.execute(Mongo.java:781) ~[stormjar.jar:?]
at com.mongodb.Mongo$2.execute(Mongo.java:764) ~[stormjar.jar:?]
at com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323) ~[stormjar.jar:?]
at org.apache.storm.mongodb.common.MongoDBClient.insert(MongoDBClient.java:61) ~[stormjar.jar:?]
at org.apache.storm.mongodb.bolt.MongoInsertBolt.execute(MongoInsertBolt.java:85) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn_7953$tuple_action_fn_7955.invoke(executor.clj:728) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7874.invoke(executor.clj:464) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.disruptor$clojure_handler$reify__7390.onEvent(disruptor.clj:40) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:439) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:418) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.daemon.executor$fn_7953$fn7966$fn_8019.invoke(executor.clj:847) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.util$async_loop$fn__625.invoke(util.clj:484) [storm-core-1.0.1.jar:1.0.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

But for less number of messages it is working fine.