Major Description
Oozie by default gathers delegation tokens for the nodes defined in mapreduce.job.hdfs-servers (or oozie.launcher.mapreduce.job.hdfs-servers in case of distcp actions) and for the workflow path.
Though this implementation is good for hdfs, we dont support occasions where the job related resources, which we want to access in runtime are present on different file systems/buckets etc...
The HDFSCredentials class should be revised to handle getting tokens for different cloud storages.
The following scenarios should be addressed:
Oozie should obtain delegation token in case
- the defaultFs is cloud
- the workload.xml is in cloud
- input/output/auxiliary files referred from workflow are in cloud
- (newly introduced feature) user could define filesystem credentials for the workflow (as its done with hive/hcat etc..) -> this would allow the user to handle the situation where Oozie could not decide which tokens are needed at launch time by default and could also get tokens for different cloud storages and buckets as well
Example for credentials addition:
<credential name="aws_auth" type="filesystem">
<property>
<name>filesystem.path</name>
<value>s3a://qe-s3-bucket-mst</value>
</property>
</credential>