[NIFI-710] Enumerate available Hadoop library versions for GetHDFS and PutHDFS

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Extensions
• Labels: hadoop hdfs library
• Environment: Unix, Hadoop
• Assignee: Oleg Zhurakousky
• Reporter: Nathan Gough
• Created: 22/Jun/15 12:42
• Updated: 24/May/16 14:27

Description

As far as I'm aware, it is only possible to use a single Hadoop library version per NiFi instance/cluster.

Would it be possible in some way to enumerate the available Hadoop library versions in the /lib directory and give the user the option to select which version of Hadoop cluster they are writing to or reading from per HDFS processor? The intent would be to allow a single NiFi instance/cluster to write to multiple HDFS clusters which may be running on different versions eg. QA and Prod. This eliminates the need for having multiple HDFS feeder NiFis only because the Hadoop versions are different.

I would try and figure this out my self however I don't know a whole lot about class loading in NiFi or generally how I would start this.