[APEXMALHAR-2557] TimeBasedDedupOperator fails with NullPointer

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.8.0
• Fix Version/s: 4.0.0
• Assignee: Unassigned
• Reporter: Vivek Bhide
• Created: 23/Mar/18 18:03
• Updated: 24/Mar/18 21:33

Description

While using the TimeBasedDedupOperator for deduping, I see that operator 
keeps failing with below NullPointer exception.

I also see that operator is always high on CPU usages. Almost reaching 100%. 
No matter the setting any values for vcores or container memory for operator

2018-03-22 15:10:10,037 INFO  stram.FSRecoveryHandler 
(FSRecoveryHandler.java:rotateLog(103)) - Creating 
hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log 
2018-03-22 15:10:10,056 INFO  stram.StreamingContainerParent 
(StreamingContainerParent.java:log(170)) - child msg: Stopped running due to 
an exception. java.lang.NullPointerException 
        at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) 
        at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) 
        at org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) 
        at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) 
        at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745) 
 context: 
PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] 
2018-03-22 15:10:10,915 WARN  stram.StreamingContainerManager 
(StreamingContainerManager.java:processOperatorFailure(1439)) - Operator 
failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 

Issue Links

• is caused by: APEXMALHAR-2492 Correct usage of empty Slice in Malhar Library