Hadoop Common
  1. Hadoop Common
  2. HADOOP-2188

RPC should send a ping rather than use client timeouts

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.1
    • Fix Version/s: 0.18.0
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Replaced timeouts with pings to check that client connection is alive. Removed the property ipc.client.timeout from the default Hadoop configuration. Removed the metric RpcOpsDiscardedOPsNum.
      Show
      Replaced timeouts with pings to check that client connection is alive. Removed the property ipc.client.timeout from the default Hadoop configuration. Removed the metric RpcOpsDiscardedOPsNum.

      Description

      Current RPC (really IPC) relies on client side timeouts to find "dead" sockets. I propose that we have a thread that once a minute (if the connection has been idle) writes a "ping" message to the socket. The client can detect a dead socket by the resulting error on the write, so no client side timeout is required. Also note that the ipc server does not need to respond to the ping, just discard it.

      1. rpc-to.patch
        27 kB
        Owen O'Malley
      2. ipc-timeout9.patch
        54 kB
        Hairong Kuang
      3. ipc-timeout8.patch
        54 kB
        Hairong Kuang
      4. ipc-timeout7.patch
        53 kB
        Hairong Kuang
      5. ipc-timeout6.patch
        51 kB
        Hairong Kuang
      6. ipc-timeout5.patch
        50 kB
        Hairong Kuang
      7. ipc-timeout4.patch
        49 kB
        Hairong Kuang
      8. ipc-timeout3.patch
        48 kB
        Hairong Kuang
      9. ipc-timeout2.patch
        47 kB
        Hairong Kuang
      10. ipc-timeout1.patch
        48 kB
        Hairong Kuang
      11. ipc-timeout.patch
        34 kB
        Hairong Kuang

        Issue Links

          Activity

          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Chansler made changes -
          Release Note This patch introduces pings when client reads are timed out. Server does not throw away old RPCs. So if clients and the server are on different versions, they are not able to function well.
          The property ipc.client.timeout is removed from the default hadoop configuration. It also removes metrics RpcOpsDiscardedOPsNum.
          Replaced timeouts with pings to check that client connection is alive. Removed the property ipc.client.timeout from the default Hadoop configuration. Removed the metric RpcOpsDiscardedOPsNum.
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Hairong Kuang made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed] [Incompatible change, Reviewed]
          Release Note This patch introduces pings when client reads are timed out. Server does not throw away old RPCs. So if clients and the server are on different versions, they are not able to function well.
          The property ipc.client.timeout is removed from the default hadoop configuration. It also removes metrics RpcOpsDiscardedOPsNum.
          Hairong Kuang made changes -
          Attachment ipc-timeout9.patch [ 12381353 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout8.patch [ 12381200 ]
          Hairong Kuang made changes -
          Hadoop Flags [Reviewed]
          Fix Version/s 0.18.0 [ 12312972 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout7.patch [ 12380967 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout6.patch [ 12380740 ]
          Owen O'Malley made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Robert Chansler made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout5.patch [ 12378398 ]
          Hairong Kuang made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.16.1 [ 12312927 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout4.patch [ 12378362 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout4.patch [ 12378353 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout4.patch [ 12378353 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout3.patch [ 12377832 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout2.patch [ 12377652 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout1.patch [ 12377562 ]
          Hairong Kuang made changes -
          Link This issue is part of HADOOP-2864 [ HADOOP-2864 ]
          Hairong Kuang made changes -
          Attachment ipc-timeout.patch [ 12375710 ]
          Robert Chansler made changes -
          Component/s dfs [ 12310710 ]
          Robert Chansler made changes -
          Assignee Owen O'Malley [ owen.omalley ] Hairong Kuang [ hairong ]
          Owen O'Malley made changes -
          Fix Version/s 0.16.0 [ 12312740 ]
          Owen O'Malley made changes -
          Assignee Owen O'Malley [ owen.omalley ]
          Owen O'Malley made changes -
          Field Original Value New Value
          Attachment rpc-to.patch [ 12369767 ]
          Owen O'Malley created issue -

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development