[HDDS-1940] Closing open container via scmcli gives false error message

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Labels: incompatibleChange pull-request-available
• Target Version/s: 0.5.0
• Assignee: Nandakumar
• Reporter: Attila Doroszlai
• Created: 09/Aug/19 15:36
• Updated: 24/Feb/20 20:55
• Resolved: 14/Nov/19 01:27

Description

scmcli close prints an error message about invalid state transition after it had successfully closed the container.

CLI

$ ozone scmcli info 2
...
Container State: OPEN
...

$ ozone scmcli close 2
...
client-09830A377AA9->f27bf787-8711-41d4-b0fd-3ef50b5c076f: receive RaftClientReply:client-09830A377AA9->f27bf787-8711-41d4-b0fd-3ef50b5c076f@group-7831D6F2EF1B, cid=0, SUCCESS, logIndex=11, commits[f27bf787-8711-41d4-b0fd-3ef50b5c076f:c12, 37ba33fe-c9ed-4ac2-a6e5-57ce658168b4:c11, feb68ba4-0a8a-4eda-9915-7dc090e5f46c:c11]
Failed to update container state #2, reason: invalid state transition from state: CLOSED upon event: CLOSE.

$ ozone scmcli info 2
...
Container State: CLOSED
...

logs

scm_1  | 2019-08-09 15:15:01 [IPC Server handler 1 on 9860] INFO  SCMClientProtocolServer:366 - Object type container id 1 op close new stage begin
dn3_1  | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO  Container:356 - Container 1 is closed with bcsId 3.
dn1_1  | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO  Container:356 - Container 1 is closed with bcsId 3.
scm_1  | 2019-08-09 15:15:02 [EventQueue-IncrementalContainerReportForIncrementalContainerReportHandler] INFO  IncrementalContainerReportHandler:176 - Moving container #1 to CLOSED state, datanode feb68ba4-0a8a-4eda-9915-7dc090e5f46c{ip: 10.5.1.6, host: ozone-static_dn3_1.ozone-static_net, networkLocation: /default-rack, certSerialId: null} reported CLOSED replica.
dn2_1  | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO  Container:356 - Container 1 is closed with bcsId 3.
scm_1  | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] INFO  SCMClientProtocolServer:366 - Object type container id 1 op close new stage complete
scm_1  | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] ERROR ContainerStateManager:335 - Failed to update container state #1, reason: invalid state transition from state: CLOSED upon event: CLOSE.
scm_1  | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] INFO  Server:2726 - IPC Server handler 3 on 9860, call Call#3 Retry#0 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.notifyObjectStageChange from 10.5.0.71:57746
scm_1  | org.apache.hadoop.hdds.scm.exceptions.SCMException: Failed to update container state #1, reason: invalid state transition from state: CLOSED upon event: CLOSE.
scm_1  | 	at org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerState(ContainerStateManager.java:336)
scm_1  | 	at org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:312)
scm_1  | 	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.notifyObjectStageChange(SCMClientProtocolServer.java:379)
scm_1  | 	at org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.notifyObjectStageChange(StorageContainerLocationProtocolServerSideTranslatorPB.java:219)
scm_1  | 	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:16398)
scm_1  | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
scm_1  | 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
scm_1  | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
scm_1  | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
scm_1  | 	at java.base/java.security.AccessController.doPrivileged(Native Method)
scm_1  | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
scm_1  | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
scm_1  | 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

Issue Links

• is duplicated by: HDDS-2383 Closing open container via SCMCli throws exception
• links to: GitHub Pull Request #153