Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.6
    • Fix Version/s: 0.94.7
    • Component/s: master, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBase-8156 is for adding support for retrying non-idempotent operations. This is useful in case NN is suffering from n/w hiccups, etc. This jira is to add similar support for 0.94.x branch.

      1. HBase-8211-v2.patch
        63 kB
        Himanshu Vashishtha
      2. HBase-8211-v1.patch
        64 kB
        Himanshu Vashishtha
      3. 8211-rebased-1.txt
        60 kB
        Lars Hofhansl
      4. 8211-rebased.txt
        60 kB
        Lars Hofhansl

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #132 (See https://builds.apache.org/job/HBase-0.94-security/132/)
          HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #132 (See https://builds.apache.org/job/HBase-0.94-security/132/ ) HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #13 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/13/)
          HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #13 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/13/ ) HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #944 (See https://builds.apache.org/job/HBase-0.94/944/)
          HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #944 (See https://builds.apache.org/job/HBase-0.94/944/ ) HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
          Hide
          Lars Hofhansl added a comment -

          Committed. Thanks for the patch Himanshu.

          Show
          Lars Hofhansl added a comment - Committed. Thanks for the patch Himanshu.
          Hide
          Lars Hofhansl added a comment -

          Last rebase had an import missing.

          Show
          Lars Hofhansl added a comment - Last rebase had an import missing.
          Hide
          Lars Hofhansl added a comment -

          Rebased patch. What I am going to commit.

          Show
          Lars Hofhansl added a comment - Rebased patch. What I am going to commit.
          Hide
          Lars Hofhansl added a comment -

          Going to commit in a bit.

          Show
          Lars Hofhansl added a comment - Going to commit in a bit.
          Hide
          Himanshu Vashishtha added a comment -

          Yes, I think it is good now.

          Show
          Himanshu Vashishtha added a comment - Yes, I think it is good now.
          Hide
          Fengdong Yu added a comment -

          It will go for commit.

          Show
          Fengdong Yu added a comment - It will go for commit.
          Hide
          Lars Hofhansl added a comment -

          Good to go?
          As I said above +1 from me.

          Show
          Lars Hofhansl added a comment - Good to go? As I said above +1 from me.
          Hide
          Himanshu Vashishtha added a comment -

          yea, did some minor fixes too.

          Show
          Himanshu Vashishtha added a comment - yea, did some minor fixes too.
          Hide
          Fengdong Yu added a comment -

          increased the log level in sleepWithRetries method in the HBaseSystem class.

          Can I fix some typo here:

          increased the log level in sleepBeforeRetry method in the HBaseFileSystem class.

          but there is a minor difference from HBase-8211-v1.patch:

          Path rootDir = new Path(TestHBaseFileSystem.conf.get(HConstants.HBASE_DIR));

          this looks good.

          Show
          Fengdong Yu added a comment - increased the log level in sleepWithRetries method in the HBaseSystem class. Can I fix some typo here: increased the log level in sleepBeforeRetry method in the HBaseFileSystem class. but there is a minor difference from HBase-8211-v1.patch: Path rootDir = new Path(TestHBaseFileSystem.conf.get(HConstants.HBASE_DIR)); this looks good.
          Hide
          Himanshu Vashishtha added a comment -

          increased the log level in sleepWithRetries method in the HBaseSystem class.

          Show
          Himanshu Vashishtha added a comment - increased the log level in sleepWithRetries method in the HBaseSystem class.
          Hide
          Fengdong Yu added a comment -

          Thanks,I will try.

          Show
          Fengdong Yu added a comment - Thanks,I will try.
          Hide
          Himanshu Vashishtha added a comment -

          s.defaultFS should be configured such as <value>hdfs://mycluster</value>, so for hbase.rootdir, can we configured such as: <value>hdfs://mycluster/hbase</value>

          Yes, it can support such path.

          Show
          Himanshu Vashishtha added a comment - s.defaultFS should be configured such as <value>hdfs://mycluster</value>, so for hbase.rootdir, can we configured such as: <value>hdfs://mycluster/hbase</value> Yes, it can support such path.
          Hide
          Fengdong Yu added a comment -

          Himanshu,

          Thanks for comments.

          <blockquote>The idea of HA support is that a client doesn't need to change any url whatsoever, otherwise its not really a failover. </blockquote>

          yes. I read the code again. that's it.

          <blockquote>If you mean a change in their value because of failover, then it is not required.</blockquote>

          for HDFS HA, fs.defaultFS should be configured such as <value>hdfs://mycluster</value>, so for hbase.rootdir, can we configured such as: <value>hdfs://mycluster/hbase</value>

          currently, we just using a specified name node address for hbase.rootdit.

          Show
          Fengdong Yu added a comment - Himanshu, Thanks for comments. <blockquote>The idea of HA support is that a client doesn't need to change any url whatsoever, otherwise its not really a failover. </blockquote> yes. I read the code again. that's it. <blockquote>If you mean a change in their value because of failover, then it is not required.</blockquote> for HDFS HA, fs.defaultFS should be configured such as <value>hdfs://mycluster</value>, so for hbase.rootdir, can we configured such as: <value>hdfs://mycluster/hbase</value> currently, we just using a specified name node address for hbase.rootdit.
          Hide
          Lars Hofhansl added a comment -

          Fengdong Yu, HDFS is handling the NN failover. This just makes sure that HBase can ride over the failover.
          The urls won't change during the NN failover.

          The problem here is that not all HDFS operations are idempotent during NN failover, and hence it is incumbent on an HDFS client (HBase in this case) to retry the operation.

          +1 on patch

          Show
          Lars Hofhansl added a comment - Fengdong Yu , HDFS is handling the NN failover. This just makes sure that HBase can ride over the failover. The urls won't change during the NN failover. The problem here is that not all HDFS operations are idempotent during NN failover, and hence it is incumbent on an HDFS client (HBase in this case) to retry the operation. +1 on patch
          Hide
          Himanshu Vashishtha added a comment -

          Having said that, I would also like to know your take on HA NN, and suggestions how HBase can make use of it.

          Show
          Himanshu Vashishtha added a comment - Having said that, I would also like to know your take on HA NN, and suggestions how HBase can make use of it.
          Hide
          Himanshu Vashishtha added a comment -

          Fendong,
          Yes, this patch just adds retries for non-idempotent ops. The intent of this patch to recover from the downtime when a standby namenode is becoming the active one. Currently, if HBase tries to do a non-idempotent operation (like create a file), it fails because NN is unavailable. This usually results in aborting the HBase server. This patch tries to cover that by retrying the op after sometime.

          FileSystem URI will be changed

          The idea of HA support is that a client doesn't need to change any url whatsoever, otherwise its not really a failover.

          Another, for HDFS HA, fs.defaultFS can be configured as "hdfs://mycluster, does this patch support "hbase.rootdir" configured "hdfs://mycluster/hbase ?

          Not sure I follow. If you mean whether properties fs.defaultFS and hbase.rootdir are supported or not, then they already are. Please look at HRS and MFS classes. If you mean a change in their value because of failover, then it is not required.
          Hope this helps.

          Show
          Himanshu Vashishtha added a comment - Fendong, Yes, this patch just adds retries for non-idempotent ops. The intent of this patch to recover from the downtime when a standby namenode is becoming the active one. Currently, if HBase tries to do a non-idempotent operation (like create a file), it fails because NN is unavailable. This usually results in aborting the HBase server. This patch tries to cover that by retrying the op after sometime. FileSystem URI will be changed The idea of HA support is that a client doesn't need to change any url whatsoever, otherwise its not really a failover. Another, for HDFS HA, fs.defaultFS can be configured as "hdfs://mycluster, does this patch support "hbase.rootdir" configured "hdfs://mycluster/hbase ? Not sure I follow. If you mean whether properties fs.defaultFS and hbase.rootdir are supported or not, then they already are. Please look at HRS and MFS classes. If you mean a change in their value because of failover, then it is not required. Hope this helps.
          Hide
          Fengdong Yu added a comment -

          Himanshu,

          I also reviewed the patch, but this just added reties, not support HDFS HA really.

          Since during Namenode failover, FileSystem URI will be changed, but we always try using the old "fs", I don't think this works well.

          Another, for HDFS HA, fs.defaultFS can be configured as "hdfs://mycluster, does this patch support "hbase.rootdir" configured "hdfs://mycluster/hbase ?

          Show
          Fengdong Yu added a comment - Himanshu, I also reviewed the patch, but this just added reties, not support HDFS HA really. Since during Namenode failover, FileSystem URI will be changed, but we always try using the old "fs", I don't think this works well. Another, for HDFS HA, fs.defaultFS can be configured as "hdfs://mycluster, does this patch support "hbase.rootdir" configured "hdfs://mycluster/hbase ?
          Hide
          Lars Hofhansl added a comment -

          Big but straightforward change. I wish HDFS would handle this stuff for us.

          LGTM.

          Show
          Lars Hofhansl added a comment - Big but straightforward change. I wish HDFS would handle this stuff for us. LGTM.
          Hide
          Himanshu Vashishtha added a comment -

          Lars Hofhansl: it will be great to know your take on this. Thanks.

          Show
          Himanshu Vashishtha added a comment - Lars Hofhansl : it will be great to know your take on this. Thanks.
          Hide
          Himanshu Vashishtha added a comment -

          Thanks for the review Matteo.

          do you plan to port the HBaseFileSystem and HLogFileSystem to trunk? this patch seems to cover code more than the one in HBASE-8156.

          Yes, trunk has the abstraction for fs calls, and its still moving in that direction. That's why I didn't change other places.
          I don't think that 0.94 will see those changes, so, I tried to cover up all possible cases here. I do think that HBaseFileSystem/HLogFileSystem should be good for trunk too. Shall I patch it up?

          re: Snapshot: yes, that's what i meant by resilent to failures in the above comment.

          Show
          Himanshu Vashishtha added a comment - Thanks for the review Matteo. do you plan to port the HBaseFileSystem and HLogFileSystem to trunk? this patch seems to cover code more than the one in HBASE-8156 . Yes, trunk has the abstraction for fs calls, and its still moving in that direction. That's why I didn't change other places. I don't think that 0.94 will see those changes, so, I tried to cover up all possible cases here. I do think that HBaseFileSystem/HLogFileSystem should be good for trunk too. Shall I patch it up? re: Snapshot: yes, that's what i meant by resilent to failures in the above comment.
          Hide
          Matteo Bertozzi added a comment -

          Looks good to me, (it seems to not apply cleanly to current 94, just a couple of import moved.. but can be rebased on merge)
          do you plan to port the HBaseFileSystem and HLogFileSystem to trunk? this patch seems to cover code more than the one in HBASE-8156.

          okish to not convert the snapshot code. you may have a snapshot failure but hbase knows how to handle that. so not a big deal... just retry.

          Show
          Matteo Bertozzi added a comment - Looks good to me, (it seems to not apply cleanly to current 94, just a couple of import moved.. but can be rebased on merge) do you plan to port the HBaseFileSystem and HLogFileSystem to trunk? this patch seems to cover code more than the one in HBASE-8156 . okish to not convert the snapshot code. you may have a snapshot failure but hbase knows how to handle that. so not a big deal... just retry.
          Hide
          Himanshu Vashishtha added a comment -

          jenkins is green.

          Show
          Himanshu Vashishtha added a comment - jenkins is green.
          Hide
          Himanshu Vashishtha added a comment -

          jenkins had some issue with the way it loads static members in the newly added test class... This one fixes that.

          Show
          Himanshu Vashishtha added a comment - jenkins had some issue with the way it loads static members in the newly added test class... This one fixes that.
          Hide
          Himanshu Vashishtha added a comment -

          patch for 0.94. Running a jenkins job now. The last jenkins job has one failure, which is fixed. Ran the patched version on a local set up (with hdfs running). Restarted the nn with some delay, and rs remains good.
          I left out some standalone tools (hbck, compactionTool, regionsplitter) from this change. Snapshot is also not touched, as it is resilent to failures (from a rs point of view).
          Please review.

          Show
          Himanshu Vashishtha added a comment - patch for 0.94. Running a jenkins job now. The last jenkins job has one failure, which is fixed. Ran the patched version on a local set up (with hdfs running). Restarted the nn with some delay, and rs remains good. I left out some standalone tools (hbck, compactionTool, regionsplitter) from this change. Snapshot is also not touched, as it is resilent to failures (from a rs point of view). Please review.
          Hide
          Himanshu Vashishtha added a comment -

          linking 0.95 jira.

          Show
          Himanshu Vashishtha added a comment - linking 0.95 jira.

            People

            • Assignee:
              Himanshu Vashishtha
              Reporter:
              Himanshu Vashishtha
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development