Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8782

Upgrade to block ID-based DN storage layout delays DN registration

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We have seen multiple incidents at production sites that there are long delays for DNs to register to the NN when upgrading to post 2.6 release.

      Further investigation shows that the DN is blocked when upgrading the storage layout introduced in HDFS-6482. The new storage layout requires making up to 64k directories in the underlying file system. Unfortunately the current implementation calls mkdirs() sequentially and upgrades each volume in sequential order.

      As a result, upgrading a DN with a lot of disks or with blocks that have random block ID takes a long time (usually in hours), and the DN won't register to the NN unless it finishes upgrading all the storage directory. The excessive delays confuse operations and break the assumption of rolling upgrades.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wheat9 Haohui Mai

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment