Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      HDFS-3146 and HDFS-3216 expose multiple DN interfaces to the client. In order for clients, in aggregate, to use multiple DN interfaces clients should pick different interfaces when transferring blocks. Given that we cache client <-> DN connections the policy of picking a remote interface at random for each new connection seems best (vs round robin for example). In the future we could make the client congestion aware. We could also establish multiple connections between the client and DN and therefore use multiple interfaces for a single block transfer. Both of those are out of scope for this jira.

      1. hdfs-3218.txt
        7 kB
        Eli Collins

        Activity

        Hide
        Eli Collins added a comment -

        Patch attached.

        Show
        Eli Collins added a comment - Patch attached.
        Hide
        Todd Lipcon added a comment -
        • I think it would make sense to add a utility function like DFSUtil.getRandomXferAddress(DatanodeID), since you have a lot of repetition fo the DFSUtil.getRandom().nextInt stuff. Or even make it a member function of the DatanodeID?

        Otherwise looks good.

        Show
        Todd Lipcon added a comment - I think it would make sense to add a utility function like DFSUtil.getRandomXferAddress(DatanodeID) , since you have a lot of repetition fo the DFSUtil.getRandom().nextInt stuff. Or even make it a member function of the DatanodeID? Otherwise looks good.

          People

          • Assignee:
            Eli Collins
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development