Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4584

Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0
    • None
    • None
    • Reviewed

    Description

      sometimes due to disk or some other problems, datanode takes minutes or tens of minutes to generate a block report. It causes the datanode not able to send heartbeat to NameNode every 3 seconds. In the worst case, it makes NameNode to detect a lost heartbeat and wrongly decide that the datanode is dead.

      It would be nice to have two threads instead. One thread is for scanning data directories and generating block report, and executes the requests sent by NameNode; Another thread is for sending heartbeats, block reports, and picking up the requests from NameNode. By having these two threads, the sending of heartbeats will not get delayed by any slow block report or slow execution of NameNode requests.

      Attachments

        1. 4584.brthread.5.patch
          50 kB
          Suresh Srinivas
        2. 4584.brthread.5.patch
          50 kB
          Suresh Srinivas
        3. 4584.brthread.4.patch
          49 kB
          Suresh Srinivas
        4. 4584.brthread.4.patch
          49 kB
          Suresh Srinivas
        5. 4584.brthread.4.patch
          50 kB
          Suresh Srinivas
        6. Design.pdf
          69 kB
          Suresh Srinivas
        7. Design.pdf
          65 kB
          Suresh Srinivas
        8. 4584.brthread.3.patch
          47 kB
          Suresh Srinivas
        9. 4584.brthread.3.patch
          46 kB
          Suresh Srinivas
        10. 4584.brthread.3.patch
          55 kB
          Suresh Srinivas
        11. 4584.brthread.3.patch
          54 kB
          Suresh Srinivas
        12. 4584.brthread.3.patch
          29 kB
          Suresh Srinivas
        13. 4584.brthread.2.patch
          54 kB
          Suresh Srinivas
        14. 4584.hbthread.patch
          20 kB
          Suresh Srinivas
        15. 4584.patch
          29 kB
          Suresh Srinivas
        16. 4584.patch
          18 kB
          Suresh Srinivas
        17. 4584.patch
          18 kB
          Suresh Srinivas
        18. 4584.patch
          18 kB
          Suresh Srinivas
        19. 4584.patch
          15 kB
          Suresh Srinivas
        20. 4584.patch
          16 kB
          Suresh Srinivas

        Issue Links

        is part of

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-4556 Block went missing

        • Major - Major loss of function.
        • Closed
        is related to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-770 SocketTimeoutException: timeout while waiting for channel to be ready for read

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HDFS-2379 0.20: Allow block reports to proceed without holding FSDataset lock

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed
        relates to

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-3232 Datanodes time out

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sureshms Suresh Srinivas
            hairong Hairong Kuang
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment