Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17408

Optimize NetworkTopology while sorting of block locations

    XMLWordPrintableJSON

    Details

      Description

      In NetworkTopology, I noticed that there are some hanging fruits to improve the performance.

      Inside sortByDistance, collections.shuffle is performed on the list before calling secondarySort.

      Collections.shuffle(list, r);
      if (secondarySort != null) {
        secondarySort.accept(list);
      }
      

      However, in different call sites, collections.shuffle is passed as the secondarySort to sortByDistance. This means that the shuffle is executed twice on each list.
      Also, logic wise, it is useless to shuffle before applying a tie breaker which might make the shuffle work obsolete.

      In addition, Daryn Sharp reported that:

      • topology is unnecessarily locking/unlocking to calculate the distance for every node
      • shuffling uses a seeded Random, instead of ThreadLocalRandom, which is heavily synchronized

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ahussein Ahmed Hussein
                Reporter:
                ahussein Ahmed Hussein
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m