[APEXMALHAR-2309] TimeBasedDedupOperator marks new tuples as duplicates if expired tuples exist

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.6.0
• Assignee: Francis Fernandes
• Reporter: Francis Fernandes
• Created: 19/Oct/16 10:29
• Updated: 02/Dec/16 21:11
• Resolved: 26/Oct/16 07:48

Description

The deduper marks valid tuples outside the expiry window as duplicates.

Consider the following configuration (number of buckets = 1 )

  <property>
    <name>dt.application.DedupTestApp.operator.Deduper.prop.expireBefore</name>
    <value>10</value>
  </property>
  <property>
    <name>dt.application.DedupTestApp.operator.Deduper.prop.bucketSpan</name>
    <value>10</value>
  </property>

The data piped in is :

"10",1474614305000,"Test"
"11",1474614315000,"Test"
"10",1474614325000,"Test"

The 3rd tuple is valid since it is outside of the expiry window. But it is marked as duplicate because although the first tuple although expired is still present in the Bucket.flash.

The issue happens when the expiry duration lesser than the checkpointing duration.

Issue Links

• links to: GitHub Pull Request #464