[HDDS-451] PutKey failed due to error "Rejecting write chunk request. Chunk overwrite without explicit request"

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: 0.2.1
• Component/s: Ozone Client
• Labels: alpha2
• Assignee: Shashikant Banerjee
• Reporter: Nilotpal Nandi
• Created: 13/Sep/18 12:35
• Updated: 27/Feb/19 23:48
• Resolved: 27/Feb/19 23:47

Description

steps taken :

------------------

1. Ran Put Key command to write 50GB data. Put Key client operation failed after 17 mins.

error seen  ozone.log :

------------------------------------

 

2018-09-13 12:11:53,734 [ForkJoinPool.commonPool-worker-20] DEBUG (ChunkManagerImpl.java:85) - writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1 chunk stage:COMMIT_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1 tmp chunk file
2018-09-13 12:11:56,576 [pool-3-thread-60] DEBUG (ChunkManagerImpl.java:85) - writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 chunk stage:WRITE_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 tmp chunk file
2018-09-13 12:11:56,739 [ForkJoinPool.commonPool-worker-20] DEBUG (ChunkManagerImpl.java:85) - writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 chunk stage:COMMIT_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 tmp chunk file
2018-09-13 12:12:21,410 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:148) - Executing cycle Number : 206
2018-09-13 12:12:51,411 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:148) - Executing cycle Number : 207
2018-09-13 12:12:53,525 [BlockDeletingService#1] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers.
2018-09-13 12:12:55,048 [Datanode ReportManager Thread - 1] DEBUG (ContainerSet.java:191) - Starting container report iteration.
2018-09-13 12:13:02,626 [pool-3-thread-1] ERROR (ChunkUtils.java:244) - Rejecting write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216}
2018-09-13 12:13:03,035 [pool-3-thread-1] INFO (ContainerUtils.java:149) - Operation: WriteChunk : Trace ID: 54834b29-603d-4ba9-9d68-0885215759d8 : Message: Rejecting write chunk request. OverWrite flag required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED
2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] ERROR (ChunkUtils.java:244) - Rejecting write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216}
2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] INFO (ContainerUtils.java:149) - Operation: WriteChunk : Trace ID: 54834b29-603d-4ba9-9d68-0885215759d8 : Message: Rejecting write chunk request. OverWrite flag required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED
 

 

Issue Links

• relates to: RATIS-489 Server RetryCache should not use size-based eviction
• relates to: HDDS-1026 Reads should fail over to alternate replica