[CAMEL-20958] camel-aws - Kinesis consumer loops on closed shards

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.6.0
• Fix Version/s: 4.8.0
• Component/s: camel-aws2-kinesis

Environment

Camel 4.6.0

Localstack 3.0.2

Reproducer attached.

• Estimated Complexity: Unknown
• Assignee: Claus Ibsen
• Reporter: Mark Wimpory
• Created: 10/Jul/24 07:16
• Updated: 21/Aug/24 06:23
• Resolved: 21/Aug/24 06:23

Description

When shards are merged the consumer reads any remaining messages from those shards and then notes that the shard has closed.  However it continuously loops as in the log exert below.  This continues on a restart.

Note this occurs no matter what type of failure option (silent, fail etc)

I have not tested so far what happens when the shard reaches its expired state (due to a localstack bug)

 
2024-07-09 12:01:43 [ForkJoinPool.commonPool-worker-4] WARN o.a.c.c.a.k.Kinesis2Consumer - The shard with id=shardId-000000000000 on stream ingestion-death-stream reached CLOSE status
2024-07-09 12:01:43 [ForkJoinPool.commonPool-worker-3] WARN o.a.c.c.a.k.Kinesis2Consumer - The shard with id=shardId-000000000001 on stream ingestion-death-stream reached CLOSE status
2024-07-09 12:01:43 [ForkJoinPool.commonPool-worker-4] INFO o.a.c.c.a.k.c.KinesisDefaultResumeAdapter - Resuming from offset 49653759514412492265585802359002135113225509352150925314 for key ingestion-death-stream
2024-07-09 12:01:43 [ForkJoinPool.commonPool-worker-3] INFO o.a.c.c.a.k.c.KinesisDefaultResumeAdapter - Resuming from offset 49653759514434793010784332982141252979858928386588016658 for key ingestion-death-stream
2024-07-09 12:01:43 [ForkJoinPool.commonPool-worker-4] DEBUG o.a.c.c.a.k.Kinesis2Consumer - Obtained new ShardIterator AAAAAAAAAAFhGdwQ2Ac96sw4ScfsOEaTn+V+4vvB9RE9cnHLqD+s4rLMSSioedJLXVbHf47GfSV8ktk53sJmQXMFb+EZOOA/PaByEh66RTC7iBK+8BCZrrL5wb4wNHV7M1qtua1Cu7NrsqEZli3+HgagoG4V7bopK5xPcezzfDN9EWclJRCnlq3Z9nJRSmB4JCmIweWsX8VrM7SHnaVabQ3Kd8l9zoj7 for shard shardId-000000000000 on stream ingestion-death-stream
2024-07-09 12:01:43 [ForkJoinPool.commonPool-worker-3] DEBUG o.a.c.c.a.k.Kinesis2Consumer - Obtained new ShardIterator AAAAAAAAAAEGg3GnGpu+nfbhufxJ7/V1iSjlD7G24d03xeQATtF9pV1dK21qLxxKpdS8g/CLPdV3MZaJaZoXkthM0J4xNvLOAkCwPpEsl9ympInzr/3QgNqFHqhGopNBlGKMyCdQUYi0UErBuUJ8dPKCq+IYRNVQvhSGg4fF4zXAlDi7jsXC7HZ2X9Xo3Q4XF/E7yEhi1YnFKAmMOmzstDkUtJd+U3MO for shard shardId-000000000001 on stream ingestion-death-stream
 

Issue Links

• links to: GitHub Pull Request #15230