Thanks for the helpful comments Tsz Wo Nicholas Sze, Arpit Agarwal, and Sanjay Radia! Really appreciate the discussion.
Rollback must always be allowed for any feature.
Sorry I used "rollback" when I meant "downgrade". With the change, downgrade won't be allowed; I will update the patch to bump the NN layout version. Rollback will work fine.
will this fix break the EZ trash support introduced by
No it won't break Trash support:
The deleted encrypted files will remain encrypted and be moved to .Trash subdirectory under the root of the encryption zone
With nested EZs, the "root of the EZ" will be the nearest ancestor with an EZ setting. I will extend the
HDFS-8831 unit test to demonstrate this.
The main motivation for nested EZ is root + subdirs as per Andrew's comment.
Yes this is the main motivation.
Is it such a big deal for an admin to set up EZ as he creates the directories in dirs?
We have received many requests from admins for this support. I think this is a natural trend as the encryption feature matures and people use it in more sophisticated ways.
I think nested encryption will complicate things like volumes
I look forward to more discussions under HDFS-8888. Meanwhile, as Andrew commented , the complexity of directory-level EZ has already been implemented. The additional complexity introduced by this change is really minimum: it basically only relaxes one if condition check.
From the ease-of-administration perspective, I think even with nested EZs (and other nested policies like erasure coding), we can always enforce volume-level uniformity. E.g. we can just disallow creating EZ (or setting EC policy) on a file/dir in a volume.