updated patch based on Eli's feedback (thx!!)
Follow ups on the non-integrated comments.
- Can you re-use an existing mechanism, eg use HADOOP_CONF_DIR by default and only require the configuration if that's not present?
TUCU: No we cannot, the current mechanism relies on having the HADOOP_CONF_DIR in the classpath. HttpFS does not have HADOOP_CONF_DIR in the classpath (furthermore, it can run in a setup where there is no other hadoop component installed)
- Are core-site.xml and hdfs-site.xml the only files that should be pulled in, eg how about hadoop-policy.xml?
TUCU: was not aware of the hadoop-policy.xml but how would the be used in HttpFS context?
- Do we need the FILE_SYSTEM_SERVICE_CREATED check? Ie if the proxy is setup correctly can users ever end up with a conf that doesn't have the hadoop conf set. If so should be a one-time initialization level check then a check performed on each call to execute.
This check is to prevent httpfs code (got bitten by this) to create and use a Configuration object without the hadoop confs in it (per first bullet item)
- If a user gets an HO4 error it means "hdfs.configuration.dir" was not defined, so worth including that in the message.
If hdfs.configuration.dir is not defined HttpFS conf dir is used. H04 is for other thing (when a Configuration object not created by FSAS is used, see previous bullet item).
On a related node is there a test that covers httpfs working w/ a non-HDFS Hadoop filesystem? Would be good to have a test that uses httpfs with local file system to catch the introduction of HDFS-isms. Can punt this to another jira.
yeah, good idea, will do.