HBase
  1. HBase
  2. HBASE-3773

Set ZK max connections much higher in 0.90

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.90.2
    • Fix Version/s: 0.90.3
    • Component/s: None
    • Labels:
      None
    • Release Note:
      The max connections per IP for ZK is now set at 2000, but only for 0.90

      Description

      I think by now we can all acknowledge that 0.90 has an issue with ZK connections, in that we create too many of them and it's also too easy for our users to shoot themselves in the foot.

      For 0.90.3, I think we should change the default configuration of 30 that we ship with and set it much much higher, I'm thinking of 32k.

        Issue Links

          Activity

          Hide
          stack added a comment -

          OK. Will be interesting to see what kinda issues folks run into when they have 32k connections to an ensemble..... Hopefully they will level off well before we hit this big number.

          Show
          stack added a comment - OK. Will be interesting to see what kinda issues folks run into when they have 32k connections to an ensemble..... Hopefully they will level off well before we hit this big number.
          Hide
          Jean-Daniel Cryans added a comment -

          I can try it.

          Show
          Jean-Daniel Cryans added a comment - I can try it.
          Hide
          Jean-Daniel Cryans added a comment -

          I tried creating as many connections as I could in the shell on my MBP using a line like this:

          3000.times { HTable.isTableEnabled("t") }

          During that time I also monitored the total number of threads used by ZK and by the shell. What happens is that the server doesn't create new threads, only the client does, and it eventually dies of OOME unable to create new threads.

          I wouldn't mind setting the max connections lower, like to a number just under MBP's default max number of threads, so that we would hit the ConnectionLoss before OOME.

          Show
          Jean-Daniel Cryans added a comment - I tried creating as many connections as I could in the shell on my MBP using a line like this: 3000.times { HTable.isTableEnabled("t") } During that time I also monitored the total number of threads used by ZK and by the shell. What happens is that the server doesn't create new threads, only the client does, and it eventually dies of OOME unable to create new threads. I wouldn't mind setting the max connections lower, like to a number just under MBP's default max number of threads, so that we would hit the ConnectionLoss before OOME.
          Hide
          Jean-Daniel Cryans added a comment -

          Committed to branch after making sure it's working as advertised.

          Show
          Jean-Daniel Cryans added a comment - Committed to branch after making sure it's working as advertised.
          Hide
          stack added a comment -

          Nice work proofing out which higher number to use.

          Show
          stack added a comment - Nice work proofing out which higher number to use.
          Hide
          Karthick Sankarachary added a comment -

          Just FYI, HBASE-3777 attempts to address the underlying ZK connection overload issue, and one of the things that it does is set the max connections per IP (viz., HConnectionManager#MAX_CACHED_HBASE_INSTANCES) to the value specified in ""hbase.zookeeper.property.maxClientCnxns". Please let me know if that's the way to go.

          Show
          Karthick Sankarachary added a comment - Just FYI, HBASE-3777 attempts to address the underlying ZK connection overload issue, and one of the things that it does is set the max connections per IP (viz., HConnectionManager#MAX_CACHED_HBASE_INSTANCES) to the value specified in ""hbase.zookeeper.property.maxClientCnxns". Please let me know if that's the way to go.
          Hide
          Jean-Daniel Cryans added a comment -

          I thought that's what it was doing before? I haven't changed that here either.

          Show
          Jean-Daniel Cryans added a comment - I thought that's what it was doing before? I haven't changed that here either.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #1888 (See https://builds.apache.org/hudson/job/HBase-TRUNK/1888/)

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #1888 (See https://builds.apache.org/hudson/job/HBase-TRUNK/1888/ )

            People

            • Assignee:
              Jean-Daniel Cryans
              Reporter:
              Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development