Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-3175

Healthy datanodes are marked as stale

    XMLWordPrintableJSON

    Details

      Description

      healthy datanodes are marked as stale due to which pipelines are not getting created

       

      scm log snippet:

      020-03-08 18:06:03,613 INFO org.apache.hadoop.hdds.scm.node.StaleNodeHandler: Datanode c03617f8-ff70-4cc6-bdf4-33441ca71471\{ip: 172.27.106.64, host: quasar-elfnqw-2.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null} moved to stale state. Finalizing its pipelines [PipelineID=dc0e7e66-c40e-4bac-86d2-3b311db482c7, PipelineID=ac390e8e-3a19-45f4-9b28-d96f1deabfca]
      2020-03-08 18:06:03,614 INFO org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Destroying pipeline:Pipeline[ Id: dc0e7e66-c40e-4bac-86d2-3b311db482c7, Nodes: c03617f8-ff70-4cc6-bdf4-33441ca71471\{ip: 172.27.106.64, host: quasar-elfnqw-2.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:ALLOCATED, leaderId:null, CreationTimestamp2020-03-08T18:01:02.964455Z]
      2020-03-08 18:06:03,614 INFO org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager: Pipeline Pipeline[ Id: dc0e7e66-c40e-4bac-86d2-3b311db482c7, Nodes: c03617f8-ff70-4cc6-bdf4-33441ca71471\{ip: 172.27.106.64, host: quasar-elfnqw-2.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:CLOSED, leaderId:null, CreationTimestamp2020-03-08T18:01:02.964455Z] moved to CLOSED state
      2020-03-08 18:06:03,620 INFO org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Destroying pipeline:Pipeline[ Id: ac390e8e-3a19-45f4-9b28-d96f1deabfca, Nodes: 2ba0ecb0-0739-4da9-9541-5fef23479f28\{ip: 172.27.138.192, host: quasar-elfnqw-7.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}bbc2192c-382e-45c9-979b-912108b7e915\{ip: 172.27.86.128, host: quasar-elfnqw-3.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}c03617f8-ff70-4cc6-bdf4-33441ca71471\{ip: 172.27.106.64, host: quasar-elfnqw-2.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:ALLOCATED, leaderId:bbc2192c-382e-45c9-979b-912108b7e915, CreationTimestamp2020-03-08T18:01:05.580596Z]
      2020-03-08 18:06:03,620 INFO org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager: Pipeline Pipeline[ Id: ac390e8e-3a19-45f4-9b28-d96f1deabfca, Nodes: 2ba0ecb0-0739-4da9-9541-5fef23479f28\{ip: 172.27.138.192, host: quasar-elfnqw-7.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}bbc2192c-382e-45c9-979b-912108b7e915\{ip: 172.27.86.128, host: quasar-elfnqw-3.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}c03617f8-ff70-4cc6-bdf4-33441ca71471\{ip: 172.27.106.64, host: quasar-elfnqw-2.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED, leaderId:bbc2192c-382e-45c9-979b-912108b7e915, CreationTimestamp2020-03-08T18:01:05.580596Z] moved to CLOSED state
      2020-03-08 18:06:06,613 INFO org.apache.hadoop.hdds.scm.node.StaleNodeHandler: Datanode 2ba0ecb0-0739-4da9-9541-5fef23479f28\{ip: 172.27.138.192, host: quasar-elfnqw-7.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null} moved to stale state. Finalizing its pipelines [PipelineID=0321db64-fa26-4c5a-a45e-59f6ab1d31c4, PipelineID=ac390e8e-3a19-45f4-9b28-d96f1deabfca]
      2020-03-08 18:06:06,613 INFO org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Destroying pipeline:Pipeline[ Id: 0321db64-fa26-4c5a-a45e-59f6ab1d31c4, Nodes: 2ba0ecb0-0739-4da9-9541-5fef23479f28\{ip: 172.27.138.192, host: quasar-elfnqw-7.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:ALLOCATED, leaderId:null, CreationTimestamp2020-03-08T18:01:05.548579Z]
      2020-03-08 18:06:06,613 INFO org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager: Pipeline Pipeline[ Id: 0321db64-fa26-4c5a-a45e-59f6ab1d31c4, Nodes: 2ba0ecb0-0739-4da9-9541-5fef23479f28\{ip: 172.27.138.192, host: quasar-elfnqw-7.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:CLOSED, leaderId:null, CreationTimestamp2020-03-08T18:01:05.548579Z] moved to CLOSED state
      2020-03-08 18:06:06,614 INFO org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Destroying pipeline:Pipeline[ Id: ac390e8e-3a19-45f4-9b28-d96f1deabfca, Nodes: 2ba0ecb0-0739-4da9-9541-5fef23479f28\{ip: 172.27.138.192, host: quasar-elfnqw-7.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}bbc2192c-382e-45c9-979b-912108b7e915\{ip: 172.27.86.128, host: quasar-elfnqw-3.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}c03617f8-ff70-4cc6-bdf4-33441ca71471\{ip: 172.27.106.64, host: quasar-elfnqw-2.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED, leaderId:bbc2192c-382e-45c9-979b-912108b7e915, CreationTimestamp2020-03-08T18:01:05.580596Z]
      2020-03-08 18:06:09,613 INFO org.apache.hadoop.hdds.scm.node.StaleNodeHandler: Datanode 9cf2c807-18d9-41bf-8abb-465cba14e26e\{ip: 172.27.82.64, host: quasar-elfnqw-9.quasar-elfnqw.root.hwx.site, networkLocation: /default-rack, certSerialId: null} moved to stale state. Finalizing its pipelines [PipelineID=62a7a71b-8f0a-43ac-8a04-b72fe0c3549d]
      

        Attachments

          Activity

            People

            • Assignee:
              ljain Lokesh Jain
              Reporter:
              nilotpalnandi Nilotpal Nandi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10m
                10m