Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4679

Datanode prints tons of log messages: Waiting for threadgroup to exit, active theads is XX

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.18.3
    • None
    • None
    • Reviewed
    • Hide
      1. Only datanode's offerService thread shutdown the datanode to avoid deadlock;
      2. Datanode checks disk in case of failure on creating a block file.
      Show
      1. Only datanode's offerService thread shutdown the datanode to avoid deadlock; 2. Datanode checks disk in case of failure on creating a block file.

    Description

      When a data receiver thread sees a disk error, it immediately calls shutdown to shutdown DataNode. But the shutdown method does not return before all data receiver threads exit, which will never happen. Therefore the DataNode gets into a dead/live lock state, emitting tons of log messages: Waiting for threadgroup to exit, active threads is XX.

      Attachments

        1. diskError3-br18.patch
          7 kB
          Hairong Kuang
        2. diskError3.patch
          9 kB
          Hairong Kuang
        3. diskError2.patch
          8 kB
          Hairong Kuang
        4. diskError1.patch
          6 kB
          Hairong Kuang
        5. diskError.patch
          6 kB
          Hairong Kuang

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4962 HADOOP-4679 to be fixed for branches >= 0.19

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-264 Better Datanode DiskOutOfSpaceException handling.

          • Major - Major loss of function.
          • Resolved

          Test - A new unit, integration or system test. HADOOP-5114 A bunch of mapred unit tests are failing on Windows

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: