[METRON-960] Pcap Topology - ArrayIndexOutOfBoundsException

• Type: Bug
• Status: To Do
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Nick Allen
• Created: 17/May/17 13:14
• Updated: 17/May/17 14:03

Description

I am running the Pcap Topology with 128 executors and 128 tasks. It is consuming packet data being landed into a Kafka topic with 128 partitions.

The topology has been running for at least 16 hours and I have been sporadically sending it ~1 gbps of traffic for periods of roughly 5 - 30 minutes.

I am sending it new packets, its counters in the Storm UI are increasing, but nothing is getting written to HDFS.

java.lang.ArrayIndexOutOfBoundsException: 4608 at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50) at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) at org.apache.hadoop.io.SequenceFile$Writer.sync(SequenceFile.java:1232) at org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1293) at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1333) at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1300) at org.apache.metron.spout.pcap.PartitionHDFSWriter.handle(PartitionHDFSWriter.java:158) at org.apache.metron.spout.pcap.HDFSWriterCallback.apply(HDFSWriterCallback.java:149) at org.apache.storm.kafka.CallbackCollector.emit(CallbackCollector.java:79) at org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:342) at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:307) at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:231) at org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)

I am also seeing other exception in the logs.

java.io.IOException: IOException flush: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/metron/pcap/pcap_pcap128_1494962815458921000_118_pcap-62-1494971709 (inode 221567): File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-100146495_67, pendingcreates: 9]
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.fsync(FSNamesystem.java:4120)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.fsync(NameNodeRpcServer.java:1334)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.fsync(ClientNamenodeProtocolServerSideTranslatorPB.java:911)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)

	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2050) ~[stormjar.jar:?]
	at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1921) ~[stormjar.jar:?]
	at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.hsync(HdfsDataOutputStream.java:95) ~[stormjar.jar:?]
	at org.apache.metron.spout.pcap.PartitionHDFSWriter$SyncHandlers$2.sync(PartitionHDFSWriter.java:71) ~[stormjar.jar:?]
	at org.apache.metron.spout.pcap.PartitionHDFSWriter.handle(PartitionHDFSWriter.java:161) ~[stormjar.jar:?]
	at org.apache.metron.spout.pcap.HDFSWriterCallback.apply(HDFSWriterCallback.java:149) [stormjar.jar:?]
	at org.apache.storm.kafka.CallbackCollector.emit(CallbackCollector.java:79) [stormjar.jar:?]
	at org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:342) [stormjar.jar:?]
	at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:307) [stormjar.jar:?]
	at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:231) [stormjar.jar:?]
	at org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
	at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]