[BEAM-1592] Unify HdfsIO and HadoopInputFormatIO

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Invalid
• Fix Version/s: Not applicable
• Component/s: io-java-hadoop-file-system, io-java-hadoop-format
• Assignee: Unassigned
• Reporter: Stephen Sisk
• Created: 02/Mar/17 18:46
• Updated: 24/Jul/20 20:13
• Resolved: 26/Feb/18 10:28

Description

HIFIO is currently in PR (https://github.com/apache/beam/pull/1994) and as per discussion in https://lists.apache.org/thread.html/803857877804165e798cf31edf079e6603eb9682b7690d52124c31e7@%3Cdev.beam.apache.org%3E, we'd like to check HIFIO in as-is, then unify the two since they share a lot of code.

[e-mail] has mentioned: "the FileInputFormat reader gets to call some special APIs that the
generic InputFormat reader cannot – so they are not completely redundant. Specifically, FileInputFormat reader can do size-based splitting."

Dan recommended: "See if we can "inline" the FileInputFormat specific parts of HdfsIO inside of HadoopInputFormatIO via reflection. If so, we can get the best of both worlds with shared code."

This seems reasonable to me.

Issue Links

• is related to: BEAM-2016 Delete HDFSFileSource/Sink