Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-1658

RaftRetryFailureException & AlreadyClosedException should not exclude pipeline from client

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Component/s: Ozone Client
    • Labels:
      None
    • Target Version/s:

      Description

      This problem can be seen at https://builds.apache.org/job/hadoop-multibranch/job/PR-846/6/testReport/org.apache.hadoop.ozone.client.rpc/TestBCSID/testBCSID/.

      As seen here, after a RaftRetryFailureException, the pipeline is excluded from the pipeline and that leads to SCM create a new pipeline. Creation of a new pipeline might not be possible in a test cluster because of limited number of nodes.

      2019-06-06 22:29:23,311 WARN  KeyOutputStream - Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.RaftRetryFailureException: Failed RaftClientRequest:client-AD0A1CB44582->73f367e6-7f91-4409-b4d3-b831e0bfb585@group-31FAD62742D6, cid=1, seq=1*, RW, org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$313/1466662004@60d08041 for 180 attempts with RetryLimited(maxAttempts=180, sleepTime=1000ms) on the pipeline Pipeline[ Id: 27d23af1-7180-42f5-b3c7-31fad62742d6, Nodes: 73f367e6-7f91-4409-b4d3-b831e0bfb585{ip: 172.17.0.2, host: 5e847226af57, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN]. The last committed block length is 0, uncommitted data length is 5 retry count 0
      2019-06-06 22:29:23,343 WARN  BlockManagerImpl - Pipeline creation failed for type:RATIS factor:ONE. Retrying get pipelines call once.
      org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot create pipeline of factor 1 using 0 nodes.
      	at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:151)
      	at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:57)
      	at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:149)
      	at org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:190)
      	at org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:172)
      	at org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:82)
      	at 
      

        Attachments

          Activity

            People

            • Assignee:
              shashikant Shashikant Banerjee
              Reporter:
              msingh Mukul Kumar Singh
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: