Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4584

Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0
    • None
    • None
    • Reviewed

    Description

      sometimes due to disk or some other problems, datanode takes minutes or tens of minutes to generate a block report. It causes the datanode not able to send heartbeat to NameNode every 3 seconds. In the worst case, it makes NameNode to detect a lost heartbeat and wrongly decide that the datanode is dead.

      It would be nice to have two threads instead. One thread is for scanning data directories and generating block report, and executes the requests sent by NameNode; Another thread is for sending heartbeats, block reports, and picking up the requests from NameNode. By having these two threads, the sending of heartbeats will not get delayed by any slow block report or slow execution of NameNode requests.

      Attachments

        1. 4584.brthread.2.patch
          54 kB
          Suresh Srinivas
        2. 4584.brthread.3.patch
          47 kB
          Suresh Srinivas
        3. 4584.brthread.3.patch
          46 kB
          Suresh Srinivas
        4. 4584.brthread.3.patch
          55 kB
          Suresh Srinivas
        5. 4584.brthread.3.patch
          54 kB
          Suresh Srinivas
        6. 4584.brthread.3.patch
          29 kB
          Suresh Srinivas
        7. 4584.brthread.4.patch
          49 kB
          Suresh Srinivas
        8. 4584.brthread.4.patch
          49 kB
          Suresh Srinivas
        9. 4584.brthread.4.patch
          50 kB
          Suresh Srinivas
        10. 4584.brthread.5.patch
          50 kB
          Suresh Srinivas
        11. 4584.brthread.5.patch
          50 kB
          Suresh Srinivas
        12. 4584.hbthread.patch
          20 kB
          Suresh Srinivas
        13. 4584.patch
          29 kB
          Suresh Srinivas
        14. 4584.patch
          18 kB
          Suresh Srinivas
        15. 4584.patch
          18 kB
          Suresh Srinivas
        16. 4584.patch
          18 kB
          Suresh Srinivas
        17. 4584.patch
          15 kB
          Suresh Srinivas
        18. 4584.patch
          16 kB
          Suresh Srinivas
        19. Design.pdf
          69 kB
          Suresh Srinivas
        20. Design.pdf
          65 kB
          Suresh Srinivas

        Issue Links

          is part of

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4556 Block went missing

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-770 SocketTimeoutException: timeout while waiting for channel to be ready for read

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HDFS-2379 0.20: Allow block reports to proceed without holding FSDataset lock

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3232 Datanodes time out

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              sureshms Suresh Srinivas
              hairong Hairong Kuang
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: