Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.89.20100924
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None

      Description

      This happened before and I am not sure how the new Master improves on it (this stuff is only available between the lines are buried in some peoples heads - one other thing I wish was for a better place to communicate what each path improves). Just so we do not miss it, there is an issue that sometimes disabling large tables simply times out and the table gets stuck in limbo.

      From the CDH User list:

      On Fri, Jan 7, 2011 at 1:57 PM, Sean Sechrist <ssechrist@gmail.com> wrote:
      To get them out of META, you can just scan '.META.' for that table name, and delete those rows. We had to do that a few months ago.

      -Sean

      That did it. For the benefit of others, here's code. Beware the literal table names, run at your own peril.

      import java.io.IOException;
      
      import org.apache.hadoop.conf.Configuration;
      import org.apache.hadoop.hbase.HBaseConfiguration;
      import org.apache.hadoop.hbase.client.HTable;
      import org.apache.hadoop.hbase.client.Delete;
      import org.apache.hadoop.hbase.client.Result;
      import org.apache.hadoop.hbase.client.MetaScanner;
      import org.apache.hadoop.hbase.util.Bytes;
      
      public class CleanFromMeta {
          public static class Cleaner implements MetaScanner.MetaScannerVisitor {
              public HTable meta = null;
              public Cleaner(Configuration conf) throws IOException {
                  meta = new HTable(conf, ".META.");
              }
      
              public boolean processRow(Result rowResult) throws IOException {
                  String r = new String(rowResult.getRow());
                  if (r.startsWith("webtable,")) {
                      meta.delete(new Delete(rowResult.getRow()));
                      System.out.println("Deleting row " + rowResult);
                  }
                  return true;
              }
          }
      
          public static void main(String[] args) throws Exception {
              String tname = ".META.";
              Configuration conf = HBaseConfiguration.create();
              MetaScanner.metaScan(conf, new Cleaner(conf), 
                   Bytes.toBytes("webtable"));
          }
      }
      

      I suggest to move this into HBaseFsck. I do not like personally to have these JRuby scripts floating around that may or may not help. This should be available if a user gets stuck and knows what he is doing (they can delete from .META. anyways). Maybe a "--disable-table <tablename> --force" or so? But since disable is already in the shell we could add an "--force" there? Or add a "--delete-table <tablename>" to the hbck?

        Activity

        Hide
        Jonathan Hsieh added a comment -

        [juneng603] eventually, after region assignments are completed and the region is opened on the target RS, information is updated in the META table so that other clients can go to the proper RS.

        Show
        Jonathan Hsieh added a comment - [juneng603] eventually, after region assignments are completed and the region is opened on the target RS, information is updated in the META table so that other clients can go to the proper RS.
        Hide
        Jonathan Hsieh added a comment -

        Vamshi root and meta are special regions but regions nonetheless. They get assigned to arbitrary (possibly different) region servers, and are hit on every new client's read and write path.

        juneng603 /hbase/uassigned is where Regions-in-transitions informatin is kept. These are modified as regions are being assigned to particular region servers. They coordinate the state between the master assigning and then RS assignee.

        Show
        Jonathan Hsieh added a comment - Vamshi root and meta are special regions but regions nonetheless. They get assigned to arbitrary (possibly different) region servers, and are hit on every new client's read and write path. juneng603 /hbase/uassigned is where Regions-in-transitions informatin is kept. These are modified as regions are being assigned to particular region servers. They coordinate the state between the master assigning and then RS assignee.
        Hide
        juneng603 added a comment -

        I also got this problem when I am using CDH4(hbase 0.92.1).
        I resolved this problem by removing /hbase/unassigned directory from a zookeeper.

        In this directory, there are lots of unassigned region information.

        so, I am curious there is some relations between zk's /hbase/unassigned directory and .META.

        Show
        juneng603 added a comment - I also got this problem when I am using CDH4(hbase 0.92.1). I resolved this problem by removing /hbase/unassigned directory from a zookeeper. In this directory, there are lots of unassigned region information. so, I am curious there is some relations between zk's /hbase/unassigned directory and .META.
        Hide
        vamshi added a comment -

        Hi Lars, i have small doubt regarding Root and .META. Actually where the Root and .META. are stored in the HBase cluster? Are they stored in two different special machines in which no other regions stored?
        One more thing is, is Root and .META. distributed or they are fully present in a single system?
        please give me a clarity on it.

        Show
        vamshi added a comment - Hi Lars, i have small doubt regarding Root and .META. Actually where the Root and .META. are stored in the HBase cluster? Are they stored in two different special machines in which no other regions stored? One more thing is, is Root and .META. distributed or they are fully present in a single system? please give me a clarity on it.
        Hide
        stack added a comment -

        Moving out of 0.92. Move it back in if you think differently.

        Show
        stack added a comment - Moving out of 0.92. Move it back in if you think differently.
        Hide
        stack added a comment -

        Moving out of 0.92. Move it back in if you think differently.

        Show
        stack added a comment - Moving out of 0.92. Move it back in if you think differently.
        Hide
        stack added a comment -

        Sounds good Lars. Should send close out to any HRegions it finds still open.

        Show
        stack added a comment - Sounds good Lars. Should send close out to any HRegions it finds still open.

          People

          • Assignee:
            Unassigned
            Reporter:
            Lars George
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development