Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.8.0
-
None
-
None
Description
While using the TimeBasedDedupOperator for deduping, I see that operator
keeps failing with below NullPointer exception.
I also see that operator is always high on CPU usages. Almost reaching 100%.
No matter the setting any values for vcores or container memory for operator
2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler (FSRecoveryHandler.java:rotateLog(103)) - Creating hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log 2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent (StreamingContainerParent.java:log(170)) - child msg: Stopped running due to an exception. java.lang.NullPointerException at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) at org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) context: PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] 2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager (StreamingContainerManager.java:processOperatorFailure(1439)) - Operator failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1
Attachments
Issue Links
- is caused by
-
APEXMALHAR-2492 Correct usage of empty Slice in Malhar Library
- Closed