HBase
  1. HBase
  2. HBASE-2961

Close zookeeper when done with it (HCM, Master, and RS)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Labels:
      None

      Description

      We're not closing down zk properly, mostly in HCM. Makes for spew in zk logs and it also causes shutdown to run longer.

      1. debug.txt
        6 kB
        stack
      2. 2961.txt
        20 kB
        stack
      3. 2961-v2.txt
        34 kB
        stack
      4. 2961-v3.txt
        45 kB
        stack

        Issue Links

          Activity

          Hide
          stack added a comment -

          Here is a mix of debug and patch... Still trying to figure why zk decides to bang its head and try and reconnect to ensemble though the sessionid thats complaining seems to have been 'closed' properly.

          Show
          stack added a comment - Here is a mix of debug and patch... Still trying to figure why zk decides to bang its head and try and reconnect to ensemble though the sessionid thats complaining seems to have been 'closed' properly.
          Hide
          stack added a comment -

          This gets rid of noise from zk in standalone mode (shutdown runs faster). It breaks tests though because fix is making a Configuration per server..... and so, new HConnections need to each get root location (Previous, they all shared single connection and the single root location). Need to make the HCM use new RootRegionTracker. Cleaner.

          Show
          stack added a comment - This gets rid of noise from zk in standalone mode (shutdown runs faster). It breaks tests though because fix is making a Configuration per server..... and so, new HConnections need to each get root location (Previous, they all shared single connection and the single root location). Need to make the HCM use new RootRegionTracker. Cleaner.
          Hide
          stack added a comment -

          This patch moves HCM to use RootRegionTracker rather than do that weird setRootLocation thing we used do – there is no need for it when HCM has a ZooKeeperWatcher inside in it.

          (This patch also removes ServerConnection, and ServerConnectionManager! interfaces and replaces them with HConnection... renames TableServers as HConnectionImplementation... hopefully now noobies won't have to spend a month of sundays trying to disentangle the flow in an hbase client).

          Not all tests pass yet.

          Show
          stack added a comment - This patch moves HCM to use RootRegionTracker rather than do that weird setRootLocation thing we used do – there is no need for it when HCM has a ZooKeeperWatcher inside in it. (This patch also removes ServerConnection, and ServerConnectionManager! interfaces and replaces them with HConnection... renames TableServers as HConnectionImplementation... hopefully now noobies won't have to spend a month of sundays trying to disentangle the flow in an hbase client). Not all tests pass yet.
          Hide
          stack added a comment -
          Make it so zk shutdown is clean.  Did it by making each server have
          its own Configuration and hence its own HConnection instance, then
          in each server (and client) added HCM.deleteConnection on way out. This
          runs clean up of the HConnection.. Added to HConnection#close the shutdown
          of zk.
          
          Did other refactor while in here; removed crazy ServerConnection and
          ServerConnectionManager stuff substituting new RootRegionTracker in its
          place.
          
          Tried to fix outstanding tests.  One still failing.
          
          M src/test/java/org/apache/hadoop/hbase/TestMultiParallel.java
            Up retries for now to make tests pass.  Needs rewrite.
          M src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
            Make a new Configuration so the setting of less tries has an effect
            and we don't retry for ever and fail the test.
          M src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java
            Minor formatting.
          M src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
            Reuse predefined variable.
          M src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
            Renamed of TableServers to HCI
          M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
            Removed ServerConnection.  Use HConnection.
            Use RootRegionTracker to keep tabs on root location.
            Add logging of zk sessionid to help w/ debug
          M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
            Minor formatting.
          M src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
            Need to preserve EOF if thats what we got.
          M src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
            Minor reformatting.
          M src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
          M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
            ServerConnection is gone.  Use HConnection instead.
          M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
            Renamed TableServers as HConnectionImplementation
            Use RootRegionTracker for figuring root location. Made it so could get
            rid of some locateRegion code.
            Added clearRegionCache(tableName) override.
          M src/main/java/org/apache/hadoop/hbase/client/HConnection.java
            Added clearRegionCache(tableName)
          M src/main/java/org/apache/hadoop/hbase/client/ServerConnection.java
          M src/main/java/org/apache/hadoop/hbase/client/ServerConnectionManager.java
            Deleted crazy stuff.
          M src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
            On delete of table, don't delete connection; connection is more than 
            just a connection to an individual table.
          
          Show
          stack added a comment - Make it so zk shutdown is clean. Did it by making each server have its own Configuration and hence its own HConnection instance, then in each server (and client) added HCM.deleteConnection on way out. This runs clean up of the HConnection.. Added to HConnection#close the shutdown of zk. Did other refactor while in here; removed crazy ServerConnection and ServerConnectionManager stuff substituting new RootRegionTracker in its place. Tried to fix outstanding tests. One still failing. M src/test/java/org/apache/hadoop/hbase/TestMultiParallel.java Up retries for now to make tests pass. Needs rewrite. M src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java Make a new Configuration so the setting of less tries has an effect and we don't retry for ever and fail the test. M src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java Minor formatting. M src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Reuse predefined variable. M src/test/java/org/apache/hadoop/hbase/client/TestHCM.java Renamed of TableServers to HCI M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Removed ServerConnection. Use HConnection. Use RootRegionTracker to keep tabs on root location. Add logging of zk sessionid to help w/ debug M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java Minor formatting. M src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java Need to preserve EOF if thats what we got. M src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java Minor reformatting. M src/main/java/org/apache/hadoop/hbase/master/ServerManager.java M src/main/java/org/apache/hadoop/hbase/master/HMaster.java M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java ServerConnection is gone. Use HConnection instead. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java Renamed TableServers as HConnectionImplementation Use RootRegionTracker for figuring root location. Made it so could get rid of some locateRegion code. Added clearRegionCache(tableName) override. M src/main/java/org/apache/hadoop/hbase/client/HConnection.java Added clearRegionCache(tableName) M src/main/java/org/apache/hadoop/hbase/client/ServerConnection.java M src/main/java/org/apache/hadoop/hbase/client/ServerConnectionManager.java Deleted crazy stuff. M src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java On delete of table, don't delete connection; connection is more than just a connection to an individual table.
          Hide
          stack added a comment -

          Committed w/o review though a fairly substantial patch; lots touched but its mostly just fixing failing tests and remove of SC and SCM.

          Show
          stack added a comment - Committed w/o review though a fairly substantial patch; lots touched but its mostly just fixing failing tests and remove of SC and SCM.

            People

            • Assignee:
              stack
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development