Currently we ran into several problems of the FileInputFormat/OutputFormat in Hive.
The requirements are:
R1. We want to support HBase:
R2. We want to selectively include files based on file names: HIVE-951
R3. We want to optionally choose to recurse on the directory structure: HIVE-1083
R4. We want to pass the filter condition into the storage (very useful for HBase, and indexed data format)
R5. We want to pass the column selection information into the storage (already done as part of the RCFile, but we can do it better)
We need to structure these requirements and the code structure in a good way to make it extensible.