Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6599

2.4 addBlock is 10 to 20 times slower compared to 0.23

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 2.3.0, 2.4.0
    • 2.5.0
    • None
    • None
    • Reviewed

    Description

      From one of our busiest 0.23 clusters:

      "AddBlockAvgTime" : 0.9514711501719515
      "CreateAvgTime" : 1.7564162389174
      "CompleteAvgTime" : 1.3310406035056548
      "BlockReceivedAndDeletedAvgTime" : 0.661210005151392

      From a not-so-busy 2.4 cluster:

      "AddBlockAvgTime" : 10.083333333333334
      "CreateAvgTime" : 1.0
      "CompleteAvgTime" : 1.1111111111111112
      "BlockReceivedAndDeletedAvgTime" : 0.07692307692307694

      When the 2.4 cluster gets a moderate amount of write requests, the latency is terrible. E.g. addBlock goes upward of 60ms.

      Attachments

        Issue Links

        is broken by

        Bug - A problem which impairs or prevents the functions of the product. HDFS-5837 dfs.namenode.replication.considerLoad does not consider decommissioned nodes

        • Major - Major loss of function.
        • Closed

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            daryn Daryn Sharp Assign to me
            kihwal Kihwal Lee
            Votes:
            0 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment