Hadoop Common
  1. Hadoop Common
  2. HADOOP-7472

RPC client should deal with the IP address changes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The current RPC client implementation and the client-side callers assume that the hostname-address mappings of servers never change. The resolved address is stored in an immutable InetSocketAddress object above/outside RPC, and the reconnect logic in the RPC Connection implementation also trusts the resolved address that was passed down.

      If the NN suffers a failure that requires migration, it may be started on a different node with a different IP address. In this case, even if the name-address mapping is updated in DNS, the cluster is stuck trying old address until the whole cluster is restarted.

      The RPC client-side should detect this situation and exit or try to recover.

      Updating ConnectionId within the Client implementation may get the system work for the moment, there always is a risk of the cached address:port become connectable again unintentionally. The real solution will be notifying upper layer of the address change so that they can re-resolve and retry or re-architecture the system as discussed in HDFS-34.

      For 0.20 lines, some type of compromise may be acceptable. For example, raise a custom exception for some well-defined high-impact upper layer to do re-resolve/retry, while other will have to restart. For TRUNK, the HA work will most likely determine what needs to be done. So this Jira won't cover the solutions for TRUNK.

      1. addr_change_dfs_0_20s.patch.txt
        2 kB
        Kihwal Lee
      2. addr_change_dfs_0_20s-1.patch.txt
        2 kB
        Kihwal Lee
      3. addr_change_dfs_0_20s-2.patch.txt
        2 kB
        Kihwal Lee
      4. addr_change_dfs_trunk.patch.txt
        3 kB
        Kihwal Lee
      5. addr_change_dfs_trunk-1.patch.txt
        3 kB
        Kihwal Lee
      6. addr_change_dfs_trunk-2.patch.txt
        2 kB
        Kihwal Lee
      7. addr_change_dfs_trunk-3.patch.txt
        2 kB
        Kihwal Lee
      8. addr_change_dfs.patch.txt
        20 kB
        Kihwal Lee
      9. addr_change_dfs-1.patch.txt
        18 kB
        Kihwal Lee
      10. addr_change_dfs-2.patch.txt
        3 kB
        Kihwal Lee
      11. addr_change_dfs-3.patch.txt
        2 kB
        Kihwal Lee

        Issue Links

          Activity

          Kihwal Lee created issue -
          Allen Wittenauer made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-34 [ HDFS-34 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs.patch.txt [ 12487564 ]
          Kihwal Lee made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.20.205.0 [ 12316390 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs-1.patch.txt [ 12487718 ]
          Kihwal Lee made changes -
          Link This issue depends on MAPREDUCE-1824 [ MAPREDUCE-1824 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs-2.patch.txt [ 12488470 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs-3.patch.txt [ 12489107 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs_trunk.patch.txt [ 12489128 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs_0_20s.patch.txt [ 12489189 ]
          Attachment addr_change_dfs_trunk-1.patch.txt [ 12489190 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs_0_20s-1.patch.txt [ 12489253 ]
          Attachment addr_change_dfs_trunk-2.patch.txt [ 12489254 ]
          Kihwal Lee made changes -
          Link This issue depends on MAPREDUCE-1824 [ MAPREDUCE-1824 ]
          Kihwal Lee made changes -
          Link This issue blocks MAPREDUCE-2764 [ MAPREDUCE-2764 ]
          Daryn Sharp made changes -
          Link This issue relates to HADOOP-7510 [ HADOOP-7510 ]
          Kihwal Lee made changes -
          Link This issue blocks MAPREDUCE-2764 [ MAPREDUCE-2764 ]
          Kihwal Lee made changes -
          Attachment addr_change_dfs_0_20s-2.patch.txt [ 12489741 ]
          Attachment addr_change_dfs_trunk-3.patch.txt [ 12489742 ]
          Suresh Srinivas made changes -
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315569 ]
          Suresh Srinivas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Kihwal Lee made changes -
          Link This issue relates to HADOOP-7492 [ HADOOP-7492 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Kihwal Lee
              Reporter:
              Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development