Type: New Feature
Affects Version/s: None
Fix Version/s: None
Currently it is not possible to append to avro files that were written and closed.
Here is a Scott Carey's reply on the mailing list:
It is not possible without modifying DataFileWriter. Please open a JIRA
It could not simply append to an OutputStream, since it must either:
- Seek to the start to validate the schemas match and find the sync
- Trust that the schemas match and find the sync marker from the last block
DataFileWriter cannot refer to Hadoop classes such as FileSystem, but we
could add something to the mapred module that takes a Path and FileSystem
something that implemements an interface that DataFileWriter can append
to. This would be something that is both a
and an OutputStream, or has both an InputStream from the start of the
existing file and an OutputStream at the end.