Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3148

The client should be able to use multiple local interfaces for data transfer

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      HDFS-3147 covers using multiple interfaces on the server (Datanode) side. Clients should also be able to utilize multiple local interfaces for outbound connections instead of always using the interface for the local hostname. This can be accomplished with a new configuration parameter (dfs.client.local.interfaces) that accepts a list of interfaces the client should use. Acceptable configuration values are the same as the dfs.datanode.available.interfaces parameter. The client binds its socket to a specific interface, which enables outbound traffic to use that interface. Binding the client socket to a specific address is not sufficient to ensure egress traffic uses that interface. Eg if multiple interfaces are on the same subnet the host requires IP rules that use the source address (which bind sets) to select the destination interface. The SO_BINDTODEVICE socket option could be used to select a specific interface for the connection instead, however it requires JNI (is not in Java's SocketOptions) and root access, which we don't want to require clients have.

      Like HDFS-3147, the client can use multiple local interfaces for data transfer. Since the client already cache their connections to DNs choosing a local interface at random seems like a good policy. Users can also pin a specific client to a specific interface by specifying just that interface in dfs.client.local.interfaces.

      This change was discussed in HADOOP-6210 a while back, and is actually useful/independent of the other HDFS-3140 changes.

      Attachments

        1. hdfs-3148.txt
          11 kB
          Eli Collins
        2. hdfs-3148.txt
          11 kB
          Eli Collins
        3. hdfs-3148.txt
          10 kB
          Eli Collins
        4. hdfs-3148-b1.txt
          21 kB
          Eli Collins
        5. hdfs-3148-b1.txt
          20 kB
          Eli Collins

        Issue Links

          Activity

            People

              eli Eli Collins
              eli2 Eli Collins
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: