HBase
  1. HBase
  2. HBASE-4025

Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descriptors

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      2011-06-23 21:39:52,524 WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl@2f56f920 appears to have been leaked
      2011-06-23 21:40:06,465 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
      java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt
      at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
      at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
      at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
      at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138)
      2011-06-23 21:40:26,790 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
      java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt
      at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
      at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
      at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
      at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138)

        Activity

        Hide
        Lars Hofhansl added a comment -

        This was committed, marking it such.

        Show
        Lars Hofhansl added a comment - This was committed, marking it such.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #1995 (See https://builds.apache.org/job/HBase-TRUNK/1995/)

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #1995 (See https://builds.apache.org/job/HBase-TRUNK/1995/ )
        Hide
        stack added a comment -

        Just to say that what Gary remarks is indeed the intent of the '.' prefix naming of log dirs, etc. This patch has hard-coding of dirs to exclude when we could have done an FS filter that skips all dirs that begin '.'. No harm. Next time through we'll change it.

        Show
        stack added a comment - Just to say that what Gary remarks is indeed the intent of the '.' prefix naming of log dirs, etc. This patch has hard-coding of dirs to exclude when we could have done an FS filter that skips all dirs that begin '.'. No harm. Next time through we'll change it.
        Hide
        Ted Yu added a comment -

        Integrated to TRUNK after renaming hbaseNonTableDirs.

        Thanks for the patch Subbu.

        Show
        Ted Yu added a comment - Integrated to TRUNK after renaming hbaseNonTableDirs. Thanks for the patch Subbu.
        Hide
        Gary Helmling added a comment -

        (repeating above but with formatting fixed)...

        Since we already have rules for valid user table names, why not just apply those in the directory listing?

        User tables are not allowed to start with '.' or '-', so ignore directory entries beginning with those. Special case '.META.' and '-ROOT-', since that's what we do most places for those 2 tables anyway.

        We already generally are following a convention of "system" directories starting with '.', so this seems sufficient to me. No need to move anything around.

        Show
        Gary Helmling added a comment - (repeating above but with formatting fixed)... Since we already have rules for valid user table names, why not just apply those in the directory listing? User tables are not allowed to start with '.' or '-', so ignore directory entries beginning with those. Special case '.META.' and '-ROOT-', since that's what we do most places for those 2 tables anyway. We already generally are following a convention of "system" directories starting with '.', so this seems sufficient to me. No need to move anything around.
        Hide
        Gary Helmling added a comment -

        Since we already have rules for valid user table names, why not just apply those in the directory listing?

        User tables are not allowed to start with '.' or '', so ignore directory entries beginning with those. Special case '.META.' and '-ROOT', since that's what we do most places for those 2 tables anyway.

        We already generally are following a convention of "system" directories starting with '.', so this seems sufficient to me. No need to move anything around.

        Show
        Gary Helmling added a comment - Since we already have rules for valid user table names, why not just apply those in the directory listing? User tables are not allowed to start with '.' or ' ', so ignore directory entries beginning with those. Special case '.META.' and '-ROOT ', since that's what we do most places for those 2 tables anyway. We already generally are following a convention of "system" directories starting with '.', so this seems sufficient to me. No need to move anything around.
        Hide
        Subbu M Iyer added a comment -

        Agree regarding the naming of variable to HBASE_NON_USER_TABLE_DIRS and as far as where it should go, I don't have a strong opinion either.

        As far as other issue is concerned, we can go either way as long as we have unique way of identifying all the user tables.

        Show
        Subbu M Iyer added a comment - Agree regarding the naming of variable to HBASE_NON_USER_TABLE_DIRS and as far as where it should go, I don't have a strong opinion either. As far as other issue is concerned, we can go either way as long as we have unique way of identifying all the user tables.
        Hide
        Ted Yu added a comment -

        Overall +1
        I am running test suite.

        Minor comment, maybe there is a better place for hbaseNonTableDirs but I don't have strong opinion.
        In HConstants, all constants are in upper case. How about renaming it to HBASE_NON_USER_TABLE_DIRS ?

        Show
        Ted Yu added a comment - Overall +1 I am running test suite. Minor comment, maybe there is a better place for hbaseNonTableDirs but I don't have strong opinion. In HConstants, all constants are in upper case. How about renaming it to HBASE_NON_USER_TABLE_DIRS ?
        Hide
        Ted Yu added a comment -

        A third option
        /hbase/System/.logs, .oldlogs, .corrupt et al.
        /hbase/<user tables>

        Users/developers are used to the current hdfs structure. This would introduce relatively small impact to existing user tables.

        Let's see what other developers think.

        Show
        Ted Yu added a comment - A third option /hbase/System/.logs, .oldlogs, .corrupt et al. /hbase/<user tables> Users/developers are used to the current hdfs structure. This would introduce relatively small impact to existing user tables. Let's see what other developers think.
        Hide
        Subbu M Iyer added a comment -

        May be we should create all tables under /hbase/tables/<table name> instead of /hbase/<table name> so that we can avoid future cases where we create other folder under /hbase such as .logs,.corrupt et al that does not contain table descriptors?

        So, this pattern will have some thing like:

        /hbase/.logs
        /hbase/.corrupt
        /hbase/.oldlogs
        /hbase/.META.
        /hbase/ROOT
        /hbase/<future non user system folders>
        /hbase/UserTables/<user table folder>/.tableinfo

        and when we need to retrieve all the table descriptors we simply iterate over the /hbase/UserTables folder rather than the /hbase and ignore all system folders.

        Other option would be:
        /hbase/System/.logs, .oldlogs, .corrupt et al.
        /hbase/UserTables/<user tables>

        This way we can avoid adding a band-aid fix to this read table descriptor logic every time we have a new system folder.

        thoughts?

        Show
        Subbu M Iyer added a comment - May be we should create all tables under /hbase/tables/<table name> instead of /hbase/<table name> so that we can avoid future cases where we create other folder under /hbase such as .logs,.corrupt et al that does not contain table descriptors? So, this pattern will have some thing like: /hbase/.logs /hbase/.corrupt /hbase/.oldlogs /hbase/.META. /hbase/ ROOT /hbase/<future non user system folders> /hbase/UserTables/<user table folder>/.tableinfo and when we need to retrieve all the table descriptors we simply iterate over the /hbase/UserTables folder rather than the /hbase and ignore all system folders. Other option would be: /hbase/System/.logs, .oldlogs, .corrupt et al. /hbase/UserTables/<user tables> This way we can avoid adding a band-aid fix to this read table descriptor logic every time we have a new system folder. thoughts?
        Hide
        Subbu M Iyer added a comment -

        First draft to ignore the .corrupt folder during getAll descriptor API call.

        Show
        Subbu M Iyer added a comment - First draft to ignore the .corrupt folder during getAll descriptor API call.

          People

          • Assignee:
            Subbu M Iyer
            Reporter:
            Subbu M Iyer
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 2h
              2h
              Remaining:
              Remaining Estimate - 2h
              2h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development