FileSystem#createNonRecursive() isn't implemented by many filesystems, using it would run the risk of hitting implementations that don't.
HBase does not have to work on every file system. We should be ok to list a support for atomic createNonRecursive() as a requirement from the underlying FS for HBase.
Is there any barrier to using a check for the parent dir existing before calling create?
I am afraid no. HBase needs this to be atomic in the namespace, since we use it for fencing. Similar to atomic renames, it is a requirement from FS, and I don't think this is unrealistic. FS owns the namespace, and it should be able to provide this semantics.
We rely on an atomic directory rename + createNonRecursive() to ensure that the region server who is dead cannot create more WAL files and commit new data or fencing. Otherwise, there will be data loss. In an non-atomic implementation, there will be a race condition, where we rename the directory, but the fenced out server will still be able to create a new file there and continue committing data which will then be lost.