[HDDS-3965] SCM failed to start up for duplicated pipeline detected

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Labels: pull-request-available upgrade-p0
• Target Version/s: 1.0.0
• Assignee: Aravindan Vijayan
• Reporter: Sammi Chen
• Created: 15/Jul/20 12:19
• Updated: 17/Jul/20 18:53
• Resolved: 17/Jul/20 18:53

Description

SCM LOG：

2020-07-15 19:25:09,768 [main] ERROR org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SCM start failed with exception
java.io.IOException: Duplicate pipeline ID PipelineID=db5966ec-140f-48d8-b0d6-e6f2ff777a77 detected.
        at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.addPipeline(PipelineStateMap.java:89)
        at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.addPipeline(PipelineStateManager.java:53)
        at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.initializePipelineState(SCMPipelineManager.java:165)
        at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.<init>(SCMPipelineManager.java:100)
        at org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:410)
        at org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:281)
        at org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:213)
        at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:624)
        at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:144)
        at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:119)


RocksDB dump, string,
rocksdb_ldb --db=scm.db scan --column_family=pipelines

$db5966ec-140f-48d8-b0d6-e6f2ff777a77ؑ????٬??????޹? : 
?
$02d3c9b4-7972-4471-a520-fff108b8d32e
                                     10.73.33.62
                                                10.73.33.62"

RATIS?M"

/default-rack???Ƕ?????Ő???? *?71-a520-fff108b8d32e:
$db5966ec-140f-48d8-b0d6-e6f2ff777a77ؑ????٬??????޹?2
?Yf?Hذ????wzw : 
?
$02d3c9b4-7972-4471-a520-fff108b8d32e
                                     10.73.33.62
                                                10.73.33.62"

RATIS?M"

HEX:
0x0A2464623539363665632D313430662D343864382D623064362D653666326666373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB001 : 0x0AAA010A2430326433633962342D373937322D343437312D613532302D666666313038623864333265120B31302E37332E33332E36321A0B31302E37332E33332E3632220A0A05524154495310824D220F0A0A5354414E44414C4F4E4510834D322430326433633962342D373937322D343437312D613532302D6666663130386238643332653A0D2F64656661756C742D7261636BA2061508F188C9CBC7B6F2E90210AEA6E3C590FEBF90A5011001180120012A3F0A2464623539363665632D313430662D343864382D623064362D653666326666373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB00132004085A7C1E5B42E
0xDB5966EC140F48D8B0D6E6F2FF777A77 : 0x0AAC010A2430326433633962342D373937322D343437312D613532302D666666313038623864333265120B31302E37332E33332E36321A0B31302E37332E33332E3632220A0A05524154495310824D220F0A0A5354414E44414C4F4E4510834D322430326433633962342D373937322D343437312D613532302D6666663130386238643332653A0D2F64656661756C742D7261636B4800A2061508F188C9CBC7B6F2E90210AEA6E3C590FEBF90A5011001180120012A3F0A2464623539363665632D313430662D343864382D623064362D653666326666373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB0013200409DFCAF8BB52E

Issue Links

• is caused by: HDDS-3925 SCM Pipeline DB should directly use UUID bytes for key rather than rely on proto serialization for key.
• links to: GitHub Pull Request #1210