I'm doing my best here to make this a friendly technical discussion. I guess my timing here is unfortunate with the merge vote pending, but since I don't intend to vote -1, this is just me reading the design doc again and posting more comments. Please take it as such. I'll restate that I think everything in the branch so far looks great.
Arpit, I actually reviewed the previous comment history very carefully before posting, as well as the updated design document. Yes, I know that I made some of these comments before, but I brought them up in the first place because I felt like they were interesting problems related to this feature. I bring them up again because, 4 months later, I was wondering if you had any thoughts on potential solutions that could be added to the doc. It's fine if automatic migration, open files, more elaborate resource management, and additional storage types are all not in immediate scope, but I assume we'll want them in the future.
I'd also like if the design doc was reworked to reflect what's in phase 1 vs. phase 2 vs. future work. This would also save me from making comments I apparently shouldn't be making on WIP sections (e.g. 6.2, 6.4). I'll say this again too, if phase 1 is done and phase 2 is starting, it seems like the design of phase 2 is exactly what be the focus of our attention right now.
The back and forth:
I mention SCR over and over again because HBase is very interested in using SSDs, and I figured supporting one of our biggest downstream projects would be a prime use case and potentially in scope. I'd be sad if not, but it'd be good to at least gather their requirements and see how we might get there.
HDFS does not support rolling upgrades today.
Well, CDH supports rolling upgrades in some situations. ATM is working on metadata upgrade with HA enabled (
HDFS-5138) and I've seen some recent JIRAs related to rolling upgrade ( HDFS-5535), so it seems like a reasonable question. At least at the protobuf level, everything so far looks compatible, so I thought it might work as long as the handler code is compatible too.
These question were also not answered:
Do you forsee heartbeats and block reports always being combined in realistic scenarios? Or are there reasons to split it? Is there any additional overhead from splitting? Can we save any complexity by not supporting split reports? I see this on the test matrix.
b1. Have you put any thought about metrics and tooling to help users and admins debug their quota usage and issues with migrating files to certain storage types? Especially because of SCR.
Thanks Arpit. Again, this feedback isn't really merge related, it's just technical discussion. Not trying to block anything here.