Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15574

Remove unnecessary sort of block list in DirectoryScanner

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.2.2, 3.3.1, 3.4.0, 3.2.3
    • datanode
    • None
    • Reviewed

    Description

      These lines of code in DirectoryScanner#scan(), obtain a snapshot of the finalized blocks from memory, and then sort them, under the DN lock. However the blocks are stored in a sorted structure (FoldedTreeSet) and hence the sort should be unnecessary.

        final List<ReplicaInfo> bl = dataset.getFinalizedBlocks(bpid);
        Collections.sort(bl); // Sort based on blockId
      

      This Jira removes the sort, and renames the getFinalizedBlocks to getSortedFinalizedBlocks to make the intent of the method more clear.

      Also added a test, just in case the underlying block structure is ever changed to something unsorted.

      Attachments

        1. HDFS-15574.001.patch
          8 kB
          Stephen O'Donnell
        2. HDFS-15574.002.patch
          8 kB
          Stephen O'Donnell
        3. HDFS-15574.003.patch
          9 kB
          Stephen O'Donnell
        4. HDFS-15574.branch-3.2.001.patch
          9 kB
          Stephen O'Donnell
        5. HDFS-15574.branch-3.2.002.patch
          10 kB
          Stephen O'Donnell
        6. HDFS-15574.branch-3.3.001.patch
          9 kB
          Stephen O'Donnell
        7. HDFS-15574.branch-3.3.002.patch
          9 kB
          Stephen O'Donnell

        Issue Links

          Activity

            People

              sodonnell Stephen O'Donnell
              sodonnell Stephen O'Donnell
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: