Stack, sorry about this. Didn't see your comment during the holidays.
Purge versions from sub-poms if the dependency is common with other modules. Inherit from parent where you can. I found this in the consensus pom:
... for what I believe are common dependencies (Its fine to have dependencies in modules with versions if the dependency only used by the submodule)
Sure, will do.
In the consensus module, there is an hbase-default.xml also? How does it relate to the one over in hbase-common?
You can purge hadoop-1 references (unless that is how you refer to your hdfs.) We don't support it in hbase 1.0.
Others have already remarked on undoing the local copies of HColumnDescriptor, etc. You had them in here so this module could 'work' without dependency? (Kill HServerAddress!) Do we need to move stuff around in hbase so you can have minimal dependencies? All the classes you have copied local here, should we move them back up and into hbase-common module if not there already?
I didn't find hbase-default.xml in the consensus package. Yes, we are cleaning up all that dependency cruft in
Run the rat tool on your feature branch. Some of these files are still missing licenses.
Yes, we didn't add licenses to mostly any of the code. Will do this.
Nice. You have already ported over ConfigurationObserver. Good.
Hehe, I wrote the original ConfigurationObserver in 0.89-fb
We'll have to redo FetchTask, etc., as protobuf? (the swifty annotations look very nice)
In FetchTask I see in the run that it has this:
// TODO in part 2: fetch log files from the peer!
Does that mean this facility of raft – catchup I presume – is yet to be implemented or is it rather that this class gets subclassed later?
If the former, is there more to do to have full implementation?
As per Rishit, the FetchTask is for advanced offline log recovery. We aren't using that part yet. We can add more comments to highlight that.
Each regionserver stands up an instance of a QuorumClient per region?
Yes, a QuorumClient is used to replicate edits. It is the most clear usage of the protocol inside RS.
A quorum name is different from region name?
It need not be. Because the consensus module is intended to be independent of WAL, and even HBase (which it is not right now, with all the dependencies). So today the quorum is being done on the WAL, so the quorum name is the same as region name. Tomorrow, we can do a quorum on something else, in which case quorum name would be something else. It is just a unique identifier for the quorum.
We'd have to move all of the quorum communication up on to hbase rpc and pb? What you lot think? Keep swift?
I would prefer swift, because we don't have to write IDL specs for the service and data-structs. That said, if it doesn't play well with the bigger project, we should discuss this in detail.
The FSM stuff could do w/ a bit of package doc on how it works. That said, it is easy to read. (Maybe we could use these primitives elsewhere in hbase)
What do you mean by a package doc? Is it synonymous to Javadocs?
Region splits handled?
Hydrabase cannot handle region splits as of now.
We'd have to figure how to do the RMap stuff in master? And bootstrapping the cluster?
Regarding Bootstrapping, we can share some ideas. There are some basic tools to bootstrap the RMap. I am moving the RMap part to hbase-server in
HBASE-12510, since this is part of the integration of HBase + Protocol. Regarding the master part, there is a diff coming out soon. We worked on something known as HydrabaseMaster which is a thin HMaster which alters the RMap as nodes die and join the quorum.
A regionserver would have to do consensuservice?
That's correct. Coming up soon.
WHen you say 'favored_nodes' in the rmap, does that imply we need 'favored nodes' working out in apache hbase or is this just how you describe quorum members for a region?
Actually favored_nodes is used for hinting the DFS where to replicate the blocks. I believe we don't have that ability in open source HDFS right? In the next diff, we are just ignoring the favored nodes part.
Is the read-side, reading from the quorum, a patch as big as this one (smile)?
Not quite, I think the patch sizes would go down exponentially from here. This probably was the largest chunk of code.