[HDDS-3619] OzoneManager fails with IllegalArgumentException for cmdType RenameKey

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Component/s: OM HA, Ozone Manager
• Labels: Triaged
• Target Version/s: 1.0.0
• Assignee: Unassigned
• Reporter: Lokesh Jain
• Created: 19/May/20 12:06
• Updated: 22/Jun/20 16:21
• Resolved: 22/Jun/20 16:21

Description

All Ozone Manager instances on startup fail with IllegalArgumentException for command type RenameKey.

2020-05-19 01:26:32,406 WARN org.apache.ratis.grpc.server.GrpcServerProtocolService: om1: installSnapshot onError, lastRequest: om2->om1#4-t34, previous=(t:34, i:44118), leaderCommit=44118, initializing? false, entries: size=1, first=(t:34, i:44119), METADATAENTRY(c:44118): org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-05-19 01:26:33,521 ERROR org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: Terminating with exit status 1: Request cmdType: RenameKey
traceID: ""
clientId: "client-E7949F1158CC"
userInfo {
  userName: "[e-mail]"
  remoteAddress: "10.17.200.43"
  hostName: "vb0933.halxg.cloudera.com"
}
renameKeyRequest {
  keyArgs {
    volumeName: "vol1"
    bucketName: "bucket1"
    keyName: "teragen/100G-terasort-input/"
    dataSize: 0
    modificationTime: 1589872757030
  }
  toKeyName: "user/ljain/.Trash/Current/teragen/100G-terasort-input/"
}
failed with exception
java.lang.IllegalArgumentException: Trying to set updateID to 35984 which is not greater than the current value of 42661 for OMKeyInfo{volume='vol1', bucket='bucket1', key='teragen/100G-terasort-input/', dataSize='0', creationTime='1589876037688', type='RATIS', factor='ONE'}
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
        at org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:107)
        at org.apache.hadoop.ozone.om.request.key.OMKeyRenameRequest.validateAndUpdateCache(OMKeyRenameRequest.java:213)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:226)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:242)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-05-19 01:26:33,526 INFO org.apache.hadoop.ozone.om.OzoneManagerStarter: SHUTDOWN_MSG:
/************************************************************

Issue Links

• is superceded by: HDDS-3476 Use persisted transaction info during OM startup in OM StateMachine