Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11492

Hadoop configuration overrides some ipc parameters including tcpNoDelay

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.98.0, 0.99.0
    • Fix Version/s: 0.99.0, 0.98.4
    • Component/s: regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      If the Hadoop configuration is read after the HBase configuration, Hadoop's settings can override HBase's settings if the names of the settings are the same. To avoid the risk of override, HBase has renamed the following settings (by prepending 'hbase.') so that you can set them independent of your setting for Hadoop. If you do not use the HBase-specific variants, the Hadoop settings will be used.

      Old Name --> New Name
      ======================================================
      ipc.server.listen.queue.size --> hbase.ipc.server.listen.queue.size
      ipc.server.max.callqueue.size --> hbase.ipc.server.max.callqueue.size
      ipc.server.callqueue.handler.factor --> hbase.ipc.server.callqueue.handler.factor
      ipc.server.callqueue.read.share --> hbase.ipc.server.callqueue.read.share
      ipc.server.callqueue.type --> hbase.ipc.server.callqueue.type
      ipc.server.queue.max.call.delay --> hbase.ipc.server.queue.max.call.delay
      ipc.server.max.callqueue.length --> hbase.ipc.server.max.callqueue.length
      ipc.server.read.threadpool.size --> hbase.ipc.server.read.threadpool.size
      ipc.server.tcpkeepalive --> hbase.ipc.server.tcpkeepalive
      ipc.server.tcpnodelay --> hbase.ipc.server.tcpnodelay
      ipc.client.call.purge.timeout --> hbase.ipc.client.call.purge.timeout
      ipc.client.connection.maxidletime --> hbase.ipc.client.connection.maxidletime
      ipc.client.idlethreshold --> hbase.ipc.client.idlethreshold
      ipc.client.kill.max --> hbase.ipc.client.kill.max
      ipc.server.scan.vtime.weight --> hbase.ipc.server.scan.vtime.weight


      Show
      If the Hadoop configuration is read after the HBase configuration, Hadoop's settings can override HBase's settings if the names of the settings are the same. To avoid the risk of override, HBase has renamed the following settings (by prepending 'hbase.') so that you can set them independent of your setting for Hadoop. If you do not use the HBase-specific variants, the Hadoop settings will be used. Old Name --> New Name ====================================================== ipc.server.listen.queue.size --> hbase.ipc.server.listen.queue.size ipc.server.max.callqueue.size --> hbase.ipc.server.max.callqueue.size ipc.server.callqueue.handler.factor --> hbase.ipc.server.callqueue.handler.factor ipc.server.callqueue.read.share --> hbase.ipc.server.callqueue.read.share ipc.server.callqueue.type --> hbase.ipc.server.callqueue.type ipc.server.queue.max.call.delay --> hbase.ipc.server.queue.max.call.delay ipc.server.max.callqueue.length --> hbase.ipc.server.max.callqueue.length ipc.server.read.threadpool.size --> hbase.ipc.server.read.threadpool.size ipc.server.tcpkeepalive --> hbase.ipc.server.tcpkeepalive ipc.server.tcpnodelay --> hbase.ipc.server.tcpnodelay ipc.client.call.purge.timeout --> hbase.ipc.client.call.purge.timeout ipc.client.connection.maxidletime --> hbase.ipc.client.connection.maxidletime ipc.client.idlethreshold --> hbase.ipc.client.idlethreshold ipc.client.kill.max --> hbase.ipc.client.kill.max ipc.server.scan.vtime.weight --> hbase.ipc.server.scan.vtime.weight

      Description

      There is an option to set tcpNoDelay, defaulted to true, but the socket channel is actually not changed. As a consequence, the server works with nagle enabled. This leads to very degraded behavior when a single connection is shared between threads. We enter into conflicts with nagle and tcp delayed ack.

      Here is an example of performance with the PE tool plus HBASE-11491:

      oneCon     #client       sleep          exeTime (seconds)                             avg latency, sleep excluded (microseconds)
      true           1               0                31                                                     310
      false          1               0                31                                                     310
      true           2               0                50                                                      500
      false          2               0               31                                                      310
      true           2                5               488 (including 200s sleeping)               2880 
      false          2               5               246  (including 200s sleeping)               460
      

      The latency is multiple by 5 (2880 vs 460) when the connection is shared. This is the delayed ack kicking in. This can be fixed by really using tcp no delay.

      Any application sharing the tcp connection between threads has the issue.

        Attachments

        1. 11492.v3.patch
          13 kB
          Nicolas Liochon
        2. 11492.v2-0.98.patch
          11 kB
          Andrew Kyle Purtell
        3. 11492.v2.patch
          5 kB
          Nicolas Liochon
        4. 11492.v2.patch
          5 kB
          Nicolas Liochon
        5. 11492.v1.withp1.patch
          1 kB
          Nicolas Liochon
        6. 11492.v1.patch
          1 kB
          Nicolas Liochon
        7. 11492.098.addendum.v2.patch
          10 kB
          Andrew Kyle Purtell
        8. 11492.098.addendum.patch
          9 kB
          Nicolas Liochon

          Issue Links

            Activity

              People

              • Assignee:
                nkeywal Nicolas Liochon
                Reporter:
                nkeywal Nicolas Liochon
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: