HBase
  1. HBase
  2. HBASE-5415

FSTableDescriptors should handle random folders in hbase.root.dir better

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.92.0
    • Fix Version/s: 0.92.1, 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I faked an upgrade on a test cluster using our dev data so I had to distcp the data between the two clusters, but after starting up and doing the migration and whatnot the web UI didn't show any table. The reason was in the master's log:

      org.apache.hadoop.hbase.TableExistsException: No descriptor for _distcp_logs_e0ehek
      at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:164)
      at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:182)
      at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1554)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

      I don't think we need to show a full stack (just a WARN maybe), this shouldn't kill the request (still see tables in the web UI), and why is that a TableExistsException?

      1. HBASE-5415.patch
        6 kB
        Jean-Daniel Cryans

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #96 (See https://builds.apache.org/job/HBase-0.92-security/96/)
        HBASE-5415 FSTableDescriptors should handle random folders in
        hbase.root.dir better (Revision 1293041)

        Result = FAILURE
        jdcryans :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #96 (See https://builds.apache.org/job/HBase-0.92-security/96/ ) HBASE-5415 FSTableDescriptors should handle random folders in hbase.root.dir better (Revision 1293041) Result = FAILURE jdcryans : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2669 (See https://builds.apache.org/job/HBase-TRUNK/2669/)
        HBASE-5415 FSTableDescriptors should handle random folders in
        hbase.root.dir better (Revision 1293042)

        Result = SUCCESS
        jdcryans :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2669 (See https://builds.apache.org/job/HBase-TRUNK/2669/ ) HBASE-5415 FSTableDescriptors should handle random folders in hbase.root.dir better (Revision 1293042) Result = SUCCESS jdcryans : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #121 (See https://builds.apache.org/job/HBase-TRUNK-security/121/)
        HBASE-5415 FSTableDescriptors should handle random folders in
        hbase.root.dir better (Revision 1293042)

        Result = SUCCESS
        jdcryans :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #121 (See https://builds.apache.org/job/HBase-TRUNK-security/121/ ) HBASE-5415 FSTableDescriptors should handle random folders in hbase.root.dir better (Revision 1293042) Result = SUCCESS jdcryans : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #301 (See https://builds.apache.org/job/HBase-0.92/301/)
        HBASE-5415 FSTableDescriptors should handle random folders in
        hbase.root.dir better (Revision 1293041)

        Result = SUCCESS
        jdcryans :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #301 (See https://builds.apache.org/job/HBase-0.92/301/ ) HBASE-5415 FSTableDescriptors should handle random folders in hbase.root.dir better (Revision 1293041) Result = SUCCESS jdcryans : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
        Hide
        Jean-Daniel Cryans added a comment -

        Committed to branch and trunk, thanks for the review Stack.

        Show
        Jean-Daniel Cryans added a comment - Committed to branch and trunk, thanks for the review Stack.
        Hide
        stack added a comment -

        +1 on commit

        Show
        stack added a comment - +1 on commit
        Hide
        Jean-Daniel Cryans added a comment -

        I still don't understand how we can tell the different between a misc directory in wrong place and a table directory missing its .tableinfo. Both would look the same to the interrogating code I'd think?

        I agree, so we used to send TEE and as you can see in the patch it never did anything special with it. Now we keep this behavior of printing something (a WARN) and then return null, which the rest of the code handles already.

        I mean, if I had client code that had a catch of a TEE, it'd stop working, right? (I'd doubt such a thing exists so I'm not too bad on removing this)

        No, since you still have to catch the IOE somehow. Also it's not part of a public interface, only HMaster and HRegionServer use TableDescriptors

        Show
        Jean-Daniel Cryans added a comment - I still don't understand how we can tell the different between a misc directory in wrong place and a table directory missing its .tableinfo. Both would look the same to the interrogating code I'd think? I agree, so we used to send TEE and as you can see in the patch it never did anything special with it. Now we keep this behavior of printing something (a WARN) and then return null, which the rest of the code handles already. I mean, if I had client code that had a catch of a TEE, it'd stop working, right? (I'd doubt such a thing exists so I'm not too bad on removing this) No, since you still have to catch the IOE somehow. Also it's not part of a public interface, only HMaster and HRegionServer use TableDescriptors
        Hide
        stack added a comment -

        Former's HTD is null, latter gets a FNFE.

        I still don't understand how we can tell the different between a misc directory in wrong place and a table directory missing its .tableinfo. Both would look the same to the interrogating code I'd think?

        Technically no, TEE (and FNFE FWIW) are both IOEs so there's no change there. I removed TEE specifically because it isn't thrown anymore.

        I mean, if I had client code that had a catch of a TEE, it'd stop working, right? (I'd doubt such a thing exists so I'm not too bad on removing this)

        Show
        stack added a comment - Former's HTD is null, latter gets a FNFE. I still don't understand how we can tell the different between a misc directory in wrong place and a table directory missing its .tableinfo. Both would look the same to the interrogating code I'd think? Technically no, TEE (and FNFE FWIW) are both IOEs so there's no change there. I removed TEE specifically because it isn't thrown anymore. I mean, if I had client code that had a catch of a TEE, it'd stop working, right? (I'd doubt such a thing exists so I'm not too bad on removing this)
        Hide
        Jean-Daniel Cryans added a comment -

        Whats difference between miscellaneous dirs under hbase.rootdir and an actual table directory that is missing its .tableinfo file?

        Former's HTD is null, latter gets a FNFE.

        We're changing our API when we remove TEE from public methods?

        Technically no, TEE (and FNFE FWIW) are both IOEs so there's no change there. I removed TEE specifically because it isn't thrown anymore.

        Show
        Jean-Daniel Cryans added a comment - Whats difference between miscellaneous dirs under hbase.rootdir and an actual table directory that is missing its .tableinfo file? Former's HTD is null, latter gets a FNFE. We're changing our API when we remove TEE from public methods? Technically no, TEE (and FNFE FWIW) are both IOEs so there's no change there. I removed TEE specifically because it isn't thrown anymore.
        Hide
        stack added a comment -

        Whats difference between miscellaneous dirs under hbase.rootdir and an actual table directory that is missing its .tableinfo file?

        We're changing our API when we remove TEE from public methods?

        Show
        stack added a comment - Whats difference between miscellaneous dirs under hbase.rootdir and an actual table directory that is missing its .tableinfo file? We're changing our API when we remove TEE from public methods?
        Hide
        Jean-Daniel Cryans added a comment -

        This patch handles it by just printing a WARN, the side effect is that this method doesn't throw TableExistsException anymore (which didn't make sense anyway) so I cleaned up a bunch of code.

        Show
        Jean-Daniel Cryans added a comment - This patch handles it by just printing a WARN, the side effect is that this method doesn't throw TableExistsException anymore (which didn't make sense anyway) so I cleaned up a bunch of code.

          People

          • Assignee:
            Jean-Daniel Cryans
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development