Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4584

Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0
    • None
    • None
    • Reviewed

    Description

      sometimes due to disk or some other problems, datanode takes minutes or tens of minutes to generate a block report. It causes the datanode not able to send heartbeat to NameNode every 3 seconds. In the worst case, it makes NameNode to detect a lost heartbeat and wrongly decide that the datanode is dead.

      It would be nice to have two threads instead. One thread is for scanning data directories and generating block report, and executes the requests sent by NameNode; Another thread is for sending heartbeats, block reports, and picking up the requests from NameNode. By having these two threads, the sending of heartbeats will not get delayed by any slow block report or slow execution of NameNode requests.

      Attachments

        1. 4584.brthread.2.patch
          54 kB
          Suresh Srinivas
        2. 4584.brthread.3.patch
          47 kB
          Suresh Srinivas
        3. 4584.brthread.3.patch
          46 kB
          Suresh Srinivas
        4. 4584.brthread.3.patch
          55 kB
          Suresh Srinivas
        5. 4584.brthread.3.patch
          54 kB
          Suresh Srinivas
        6. 4584.brthread.3.patch
          29 kB
          Suresh Srinivas
        7. 4584.brthread.4.patch
          49 kB
          Suresh Srinivas
        8. 4584.brthread.4.patch
          49 kB
          Suresh Srinivas
        9. 4584.brthread.4.patch
          50 kB
          Suresh Srinivas
        10. 4584.brthread.5.patch
          50 kB
          Suresh Srinivas
        11. 4584.brthread.5.patch
          50 kB
          Suresh Srinivas
        12. 4584.hbthread.patch
          20 kB
          Suresh Srinivas
        13. 4584.patch
          29 kB
          Suresh Srinivas
        14. 4584.patch
          18 kB
          Suresh Srinivas
        15. 4584.patch
          18 kB
          Suresh Srinivas
        16. 4584.patch
          18 kB
          Suresh Srinivas
        17. 4584.patch
          15 kB
          Suresh Srinivas
        18. 4584.patch
          16 kB
          Suresh Srinivas
        19. Design.pdf
          69 kB
          Suresh Srinivas
        20. Design.pdf
          65 kB
          Suresh Srinivas

        Issue Links

          Activity

            People

              sureshms Suresh Srinivas
              hairong Hairong Kuang
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: