Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.94.6
-
Fix Version/s: 0.94.7
-
Component/s: master, regionserver
-
Labels:None
-
Hadoop Flags:Reviewed
Description
HBase-8156 is for adding support for retrying non-idempotent operations. This is useful in case NN is suffering from n/w hiccups, etc. This jira is to add similar support for 0.94.x branch.
Issue Links
- is related to
-
HBASE-8156
Support for Namenode HA for non-idempotent operations
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
patch for 0.94. Running a jenkins job now. The last jenkins job has one failure, which is fixed. Ran the patched version on a local set up (with hdfs running). Restarted the nn with some delay, and rs remains good.
I left out some standalone tools (hbck, compactionTool, regionsplitter) from this change. Snapshot is also not touched, as it is resilent to failures (from a rs point of view).
Please review.
jenkins had some issue with the way it loads static members in the newly added test class... This one fixes that.
jenkins is green.
Looks good to me, (it seems to not apply cleanly to current 94, just a couple of import moved.. but can be rebased on merge)
do you plan to port the HBaseFileSystem and HLogFileSystem to trunk? this patch seems to cover code more than the one in HBASE-8156.
okish to not convert the snapshot code. you may have a snapshot failure but hbase knows how to handle that. so not a big deal... just retry.
Thanks for the review Matteo.
do you plan to port the HBaseFileSystem and HLogFileSystem to trunk? this patch seems to cover code more than the one in
HBASE-8156.
Yes, trunk has the abstraction for fs calls, and its still moving in that direction. That's why I didn't change other places.
I don't think that 0.94 will see those changes, so, I tried to cover up all possible cases here. I do think that HBaseFileSystem/HLogFileSystem should be good for trunk too. Shall I patch it up?
re: Snapshot: yes, that's what i meant by resilent to failures in the above comment.
Lars Hofhansl: it will be great to know your take on this. Thanks.
Big but straightforward change. I wish HDFS would handle this stuff for us.
LGTM.
Himanshu,
I also reviewed the patch, but this just added reties, not support HDFS HA really.
Since during Namenode failover, FileSystem URI will be changed, but we always try using the old "fs", I don't think this works well.
Another, for HDFS HA, fs.defaultFS can be configured as "hdfs://mycluster, does this patch support "hbase.rootdir" configured "hdfs://mycluster/hbase ?
Fendong,
Yes, this patch just adds retries for non-idempotent ops. The intent of this patch to recover from the downtime when a standby namenode is becoming the active one. Currently, if HBase tries to do a non-idempotent operation (like create a file), it fails because NN is unavailable. This usually results in aborting the HBase server. This patch tries to cover that by retrying the op after sometime.
FileSystem URI will be changed
The idea of HA support is that a client doesn't need to change any url whatsoever, otherwise its not really a failover.
Another, for HDFS HA, fs.defaultFS can be configured as "hdfs://mycluster, does this patch support "hbase.rootdir" configured "hdfs://mycluster/hbase ?
Not sure I follow. If you mean whether properties fs.defaultFS and hbase.rootdir are supported or not, then they already are. Please look at HRS and MFS classes. If you mean a change in their value because of failover, then it is not required.
Hope this helps.
Having said that, I would also like to know your take on HA NN, and suggestions how HBase can make use of it.
Fengdong Yu, HDFS is handling the NN failover. This just makes sure that HBase can ride over the failover.
The urls won't change during the NN failover.
The problem here is that not all HDFS operations are idempotent during NN failover, and hence it is incumbent on an HDFS client (HBase in this case) to retry the operation.
+1 on patch
Himanshu,
Thanks for comments.
<blockquote>The idea of HA support is that a client doesn't need to change any url whatsoever, otherwise its not really a failover. </blockquote>
yes. I read the code again. that's it.
<blockquote>If you mean a change in their value because of failover, then it is not required.</blockquote>
for HDFS HA, fs.defaultFS should be configured such as <value>hdfs://mycluster</value>, so for hbase.rootdir, can we configured such as: <value>hdfs://mycluster/hbase</value>
currently, we just using a specified name node address for hbase.rootdit.
s.defaultFS should be configured such as <value>hdfs://mycluster</value>, so for hbase.rootdir, can we configured such as: <value>hdfs://mycluster/hbase</value>
Yes, it can support such path.
Thanks，I will try.
increased the log level in sleepWithRetries method in the HBaseSystem class.
increased the log level in sleepWithRetries method in the HBaseSystem class.
Can I fix some typo here:
increased the log level in sleepBeforeRetry method in the HBaseFileSystem class.
but there is a minor difference from HBase-8211-v1.patch:
Path rootDir = new Path(TestHBaseFileSystem.conf.get(HConstants.HBASE_DIR));
this looks good.
yea, did some minor fixes too.
Good to go?
As I said above +1 from me.
It will go for commit.
Yes, I think it is good now.
Going to commit in a bit.
Rebased patch. What I am going to commit.
Last rebase had an import missing.
Committed. Thanks for the patch Himanshu.
Integrated in HBase-0.94 #944 (See https://builds.apache.org/job/HBase-0.94/944/)
HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785)
Result = SUCCESS
larsh :
Files :
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
Integrated in HBase-0.94-security-on-Hadoop-23 #13 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/13/)
HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785)
Result = FAILURE
larsh :
Files :
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
Integrated in HBase-0.94-security #132 (See https://builds.apache.org/job/HBase-0.94-security/132/)
HBASE-8211 Support for NN HA for 0.94 (Himanshu) (Revision 1464785)
Result = SUCCESS
larsh :
Files :
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/Reference.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFileSystem.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHBaseFileSystem.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
linking 0.95 jira.