Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16386

Reduce DataNode load when FsDatasetAsyncDiskService is working

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Our DataNode node has 36 disks. When FsDatasetAsyncDiskService is working, it will cause a high load on the DataNode.
      Here are some monitoring related to memory:

      Since each disk deletes the block asynchronously, and each thread allows 4 threads to work,
      This will cause some troubles to the DataNode, such as increased cpu and increased memory.
      We should appropriately reduce the number of jobs of the total thread so that the DataNode can work better.

      Attachments

        1. monitor.png
          138 kB
          JiangHua Zhu

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jianghuazhu JiangHua Zhu
            jianghuazhu JiangHua Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h 10m
                4h 10m

                Slack

                  Issue deployment